But military officials dismissed the claims of a massacre in Barwana, saying they had been fabricated.
The UN's envoy to Iraq, Nickolay Mladenov, welcomed Mr Abadi's decision.
"It is the responsibility of the government to ensure that all armed forces are under its control, that rule of law is respected and that civilians are protected in all areas of the country, including those areas recently liberated from IS," he said in a statement.
On Monday, the Iraqi military declared that it had managed to drive IS militants out of all of Diyala province after recapturing the last contested area, the district of Muqdadiya.
The operation came at a high cost, with officials saying that at least 70 soldiers and pro-government militiamen were killed and 400 others wounded in the three days of fighting. About 65 IS militants were also reportedly killed.
At the end of the operation, Shia militiamen are reported to have besieged the village of Barwana, where many residents of nearby villages had sought shelter.
Abu Omar, a businessman from Sinsil, to the south-west of Barwana, told Reuters news agency that about 10 Humvees carrying a few dozen armed men arrived in the village on Monday afternoon.
They were wearing black and brown uniforms, suggesting some were affiliated with Shia militia and government security forces, he said. Others appeared to be civilians.
Men and boys were dragged from their homes, beaten and cursed with sectarian slurs, and then led in small groups to a field, Abu Omar added.
They were then made to kneel and stare at the ground as the gunmen selected victims one by one and then led them behind a mud wall.
"They took them behind the wall. Less than a minute, then a gunshot," Abu Omar said. "All we could hear was the gunshots. We couldn't see."
After the gunmen left the village, militiamen are reported to have surrounded the village and refused to allow medical workers to enter.
A Sunni member of Diyala's provincial council, Haqqi al-Jabouri, told Reuters that at least 72 men were killed in Barwana, and that 35 others were missing and suspected detained.
However, security officials disputed the accounts, with some saying that IS militants were responsible and others that they were fabricated by IS supporters.
Shia militia have been accused of kidnapping and killing scores of Sunni civilians since Islamic State launched an offensive in northern Iraq last June that saw the jihadist group seize large swathes of the country.
Prime Minister Abadi, who took office in September, has pledged to rein in the militia and ensure their fighters are integrated into the security forces.