Diplomatic sources said discussion at a late-running summit dinner became increasingly heated and "emotional" as Renzi accused his peers of looking after only their own interests.
"If that's your idea of Europe, you can keep it," Renzi told his 27 counterparts, Italian sources told AFP.
"Either give us solidarity or don't waste our time," they quoted him as saying.
A European source said the EU's two top figures, European Council President Donald Tusk and European Commission chief Jean-Claude Juncker, had clashed at the meeting but the two rejected the suggestion.
"Don't believe those who are tweeting, leaking info. Whenever we have difficulties, we discuss these things between ourselves," Juncker told a press conference with Tusk early Friday.
What really mattered instead was that EU leaders had approved the numbers of asylum seekers to be resettled as proposed by the Commission.
"I don't care if it is to be done voluntarily or not because the most important thing was that we agreed on the 60,000," he said.
Juncker is the driving force behind proposals for 40,000 Syrian and Eritrean asylum-seekers who have already arrived in Europe to be redistributed under a compulsory system.
Another 20,000 Syrians living in camps outside Europe will also be resettled across the 28-nation bloc.
Tusk, a former Polish premier whose country opposes the quota system, said "migrants with no legal right to enter the EU will be returned" and negotiations with countries of origin would be speeded up to take them.
Member states will in the meantime agree how many asylum seekers they are willing to take on a voluntary basis by mid-July, he added.
Renzi said the outcome of the meeting fell short of what Italy needed. "Relocation is a first step but for us this is not the solution to our problems."
More than 100,000 migrants have crossed the Mediterranean so far this year, most of them landing in Italy, Greece and Malta who say their EU partners should share more of the burden.