Ezzedin Al-Awami, the first vice-president of the parliament, said that in the confusion of a vote taken by parliament Sunday, Miitig failed to get the number of votes required.
A statement posted on the government website said the Islamist-backed businessman received 113 votes but needed 120 to be declared the country's new prime minister in accordance with the constitution.
In a second statement published on the website, Awami says the vote of confidence is "void and illegal."
It followed an earlier chaotic session of the General National Congress (GNC).
After the initial vote count, GNC official Salah al-Makhzoum said Miitig had in fact clinched 121 votes in the 185-seat interim parliament, apparently after a recount, and beat challenger Omar al-Hassi, a university professor.
Some deputies denounced the recount, which they said took place after the session had officially closed.
"What is happening is illegal," said GNC spokesman Omar Hmidan.
The voting had been broadcast live on state television up until Awami closed the session, saying a new one would convene.
Miitig, 42, would be -- if the vote is confirmed -- Libya's youngest and fifth prime minister since veteran dictator Moamer Kadhafi was toppled and killed in a 2011 uprising.
Since then, successive governments in the oil-rich North African country have struggled to impose order as heavily armed former rebel brigades have carved out their own fiefdoms and refused to join the security forces.
The job of prime minister has proven challenging and dangerous -- Ali Zeidan, who was voted out by parliament for failing to prevent a rebel oil shipment in March, was kidnapped by gunmen last year and held for several hours before being released.
Last month, Zeidan's defence minister Abdullah al-Thani was appointed to replace him, but stepped down after just five days, saying he and his family had come under attack.
State television broadcast chaotic footage from Sunday's session.
At first, Awami said Miitig defeated university professor Omar al-Hassi by 73-43 votes but mustered only 113 of the 120 votes required for a vote of confidence.
But the second vice president, Salah al-Makhzoum, who ranks lower than Awami, later announced that the relatively unknown businessman had clinched 121 votes in the 185-seat interim parliament, apparently after a recount.
Another chaotic vote
Miitig -- who has vowed to rebuild state institutions, namely the army and police -- would normally have two weeks to form a government.
But liberal lawmaker Sherif al-Wafi said he and other deputies "will take the necessary legal measures" to stop Miitig from heading the government.
Deputy Fatima al-Majbari said Makhzoum's recount included "votes of lawmakers who were absent."
It was the second time in a week that the GNC met to decide between the two candidates who were initially among a group of seven hopefuls.
After an initial meeting on April 27, parliament gathered again on Tuesday, when Miitig won 67 votes in a first round with Hassi in second place with 34 votes.
A second round of voting on Tuesday was interrupted by gunmen who stormed into parliament for reasons that are still unclear, shooting and forcing deputies to evacuate the premises.
The GNC was elected in July 2012 for an 18-month term, but in February it extended its mandate to December, sparking widespread protests.
Parliament's Islamist supporters were in favour of the extension while the liberal National Forces Alliance slammed the move as undemocratic.
The political turmoil comes as militants have carried out near-daily attacks on security forces, particularly in the restive eastern city of Benghazi, cradle of the uprising that ended Kadhafi's four-decade rule.
Some of the most formidable rebels hail from Misrata, Libya's third largest city and Miitig's hometown, which saw some of the worst fighting of the NATO-backed uprising.