Washington, which helped install Maliki following its 2003 invasion that toppled Saddam Hussein, congratulated Haidar al-Abadi, a former Maliki lieutenant who was named by President Fouad Masoum to replace him.
But Maliki's Dawa Party declared his replacement illegal, and Maliki's son-in-law said he would overturn it in court. Washington delivered a stern warning to Maliki not to "stir the waters" by using force to cling to power.
A Shi'ite Muslim Islamist, Maliki is blamed by his erstwhile allies in Washington and Tehran for driving the alienated Sunni minority into a revolt that threatens to destroy the country. Leaders of Iraq's Sunni and Kurdish communities have demanded he go, and many fellow Shi'ites have turned against him.
Maliki himself said nothing about the decision to replace him, standing in grim-faced silence on Monday next to a member of his Dawa Party, who read out a statement on national television declaring Abadi's nomination illegal.
Abadi "represents only himself", the Dawa member, Khalaf Abdul-Samad said.
Maliki's son-in-law Hussein al-Maliki told Reuters his camp would fight the "illegal" decision: "We will not stay silent."
"The nomination is illegal and a breach of the constitution. We will go to the federal court to object."
Washington made its support for the new leader clear. The White House said Vice President Joe Biden relayed President Barack Obama's congratulations to Abadi in a phone call.
"The prime minister-designate expressed his intent to form a broad-based, inclusive government capable of countering the threat of the Islamic State of Iraq and the Levant," the White House said in a statement, using a previous name for the Sunni militant group that now calls itself the Islamic State.
The new political crisis comes just days after Washington launched its first military action in Iraq since pulling its troops out in 2011. US warplanes have bombed Sunni insurgents from the Islamic State, who have marched through northern and western Iraq since June.
Washington says it is taking limited action to protect a Kurdish autonomous region and prevent what Obama called a potential "genocide" of religious minorities targeted by the militants.
The fighters made new gains against Kurdish forces despite three days of US air strikes, while Baghdad, long braced for the Sunni fighters to attack, was now tensing for possible clashes between Maliki and rivals within the Shi'ite majority.
President Masoum asked Abadi to form a government that could win the support of all groups in a parliament elected in April. In remarks broadcast on television, Masoum, a Kurd, urged Abadi to "form a broader-based government" over the next month.
Abadi urged national unity against the "barbaric" Islamic State, which has driven tens of thousands from their homes as it swept aside Baghdad's troops to consolidate a "caliphate" in Iraq and Syria.
"We all have to cooperate to stand against this terrorist campaign launched on Iraq and to stop all terrorist groups," he said in broadcast remarks after meeting Masoum.
As police and elite armed units, many equipped and trained by the United States, locked down the capital's streets, Secretary of State John Kerry aimed a stark warning at Maliki against fighting to hold on to power.
"There should be no use of force, no introduction of troops or militias in this moment of democracy for Iraq," Kerry said. "The government formation process is critical in terms of sustaining stability and calm in Iraq and our hope is that Mr. Maliki will not stir those waters.
"There will be little international support of any kind whatsoever for anything that deviates from the legitimate constitution process that is in place and being worked on now."
Under Iraq's post-Saddam governing system, designed to avert conflict by giving all groups a stake, the speaker of parliament is a Sunni and the largely ceremonial president a Kurd. Most authority is wielded by the prime minister, a Shi'ite.
Maliki's opponents accuse him of abusing the system by keeping key security posts in his own hands instead of sharing them with other groups, alienating Sunnis in particular by ordering the arrest of their political leaders. Islamic State fighters were able to exploit that resentment to win support from other Sunni armed groups.
Maliki's Shi'ite State of Law bloc emerged as the biggest group in parliament in the April election, but does not have enough seats to rule without support from Sunnis, Kurds and other Shi'ite blocs, nearly all of which demand he go.
He has nevertheless stayed on in a caretaker capacity while arguing that the constitution requires his bloc to be given the first opportunity to form a government. He has used courts before to keep power: in the previous election in 2010, when State of Law was second, a court let him form a cabinet.
A US official insisted Washington had not been involved in the selection of Abadi, but said "everybody is pretty relieved that they have chosen somebody and that it was not Maliki".
Maliki also appears to have alienated his supporters in Iran, the regional Shi'ite power, which has sent military advisers to help organise the battle against the Islamic State. Iraq's most influential Shi'ite cleric, Ali Sistani, all but ordered Maliki to leave power on Friday, declaring that politicians who cling to power were making a "grave mistake".
Obama says a more inclusive government in Baghdad is a pre-condition for more aggressive US military support against the Islamic State. He has rejected calls in some quarters for a return of US ground troops, apart from several hundred military advisers sent in June.
The Islamic State which sees Shi'ites as heretics who deserve to be killed, has ruthlessly moved through one town after another, using tanks and heavy weapons it seized from soldiers who have fled in their thousands.
On Monday, police said the fighters had seized the town of Jalawla, 115 km (70 miles) northeast of Baghdad, after driving out the forces of the autonomous Kurdish regional government.
Washington and its European allies are considering requests for more direct military aid from the Kurds, who have themselves differed with Maliki over the division of oil resources and took advantage of the Islamists' advance to expand their territory.
On Sunday, a government minister said Islamic State militants had killed hundreds of people from the small, Kurdish-speaking Yazidi religious sect, burying some alive and taking women as slaves. No confirmation was available of the killings.
Thousands of Yazidis have taken refuge in the past week on the arid heights of Mount Sinjar, close to the Syrian border. The Islamic State considers the Yazidis, who follow an ancient faith derived from Zoroastrianism, to be "devil worshippers".
The bloodshed could increase pressure on Western powers to do more to help those who have fled the Islamic State's offensive. They have already dropped supplies and US aircraft have been bombing the militants since Friday.