Diarra declared in a brief speech given at the premises of, and aired by, national broadcaster ORTM. He gave no reason for his decision.
Malian soldiers arrested Prime Minister Cheick Modibo Diarra at his home late Monday, acting on the orders of ex-coup leader Amadou Sanogo, a member of the premier's entourage said.
"The prime minister was arrested by about 20 soldiers who came from Kati" a military barracks outside Bamako and headquarters of the former putschists, said the source, who witnessed the arrest.
"They said Captain Sanogo sent them to arrest him," he added. A security source confirmed the information.
The member of Diarra's entourage, speaking on condition of anonymity, said the soldiers had "smashed in the door of the prime minister's residence and took him away a bit violently."
Diarra, a noted astrophysicist who has worked on several NASA space programs and served as Microsoft's chairperson for Africa, was earlier due to leave for Paris for a medical check-up.
He cancelled plans to head to the airport when he learned his baggage had been taken off the plane meant to take him to France.
The source said Diarra had recorded a short message which was to be broadcast on state television, but soldiers went to the broadcaster's headquarters to confiscate the tape.
Diarra was named as prime minister in an interim government just weeks after a disastrous March coup which plunged the once stable democracy into a crisis which has seen over half its territory seized by hardline Islamists.
The 60-year-old premier is a staunch advocate of plans to send a west African intervention force into the occupied territory to drive out the extremists who are running the zone according to their brutal interpretation of sharia, or Islamic, law.
Citizens have been flogged, had their hands amputated and been stoned to death as punishments for their transgressions.
The previously unknown officer Sanogo, launched a coup on March 22 to oust President Amadou Toumani Toure's government only six weeks before an election marking the end of his time in office.
The move came amid mounting anger by soldiers at their rout by Tuareg separatists who were slowly making headway in a fresh rebellion to conquer the north and declare independence for their homeland which they call Azawad.
The coup only made it easier for the rebels and their Islamist allies to seize control of an area larger than France.
However the unlikely alliance between the secular separatists and Al-Qaeda linked Islamists quickly crumbled and the Tuareg were driven out of key positions, leaving the vast arid zone in the hands of extremists.
West African nations are pressing hard for the United Nations Security Council to approve a plan for military intervention, which is being backed by France, while Germany and the United States have offered training and logistical support.
Western powers fear the north of Mali could become a new sanctuary for terrorist groups.
But misgivings are rife over the plan to send in 3,300 west African troops. Many of Mali's neighbours still prefer a negotiated solution and both the UN and US have urged caution.
US ambassador Susan Rice last week argued the west African troops would be ill-suited for the desert battle against groups such as Ansar Dine (Defenders of Faith), al Qaeda in the Islamic Maghreb (AQIM) and its offshoot the Movement for Oneness and Jihad in West Africa (MUJAO).
The US wants more details on the capabilities of the force to achieve the mission's objective, the cost of the mission, logistical needs and plan to minimise impacts on civilian security and the humanitarian situation.
The fragility of an interim government in Bamako which has failed to assert itself politically and has yet to plan fresh elections, is also a key concern as military plans come together.
Sanogo and his men have remained influential in Bamako.