Now he's back as prime minister-elect, with the army watching his every move, especially steps planned to ease tension with arch-rival India.
Sharif's Pakistan Muslim League-Nawaz (PML-N) won 124 of 272 contested seats in this month's parliamentary election.
Its nearest rival, the Pakistan People's Party, won just 31 in the first democratic handover of power since independence in 1947.
But Sharif is inheriting an almost-broke nation that has been ruled by the military for more than half the 66 years since then.
It is wracked by a Taliban insurgency, sectarian violence, daily power cuts and spiralling unemployment among its 180 million people.
"Pakistan has so many problems that he needs the help and support of everybody, including the army," said Shazad Chaudhry, a former air vice marshal.
The question is: will the army give Sharif the freedom to do what he wants?
The top source of potential friction is his aim to boost trade and ties with India.
The two nuclear-armed neighbours have fought three wars since both became independent of colonial Britain, two over the still-disputed Himalayan territory of Kashmir.
Sharif's policies towards India will be heavily scrutinised - some in the army justify its massive budget on the potential threat from India - but there are signs the military may be slightly more amenable to overtures than in the past.
In 1999, the army torpedoed Sharif's attempts to improve relations by secretly sending soldiers disguised as militants to capture Indian outposts in the heights of Kargil, in the north of Kashmir.
When Sharif tried to fire Pervez Musharraf, his then-chief of army staff, Musharraf took power in a coup.
Sharif has promised an inquiry into the Kargil invasion.
Most officers believe that is a political move aimed mostly at containing Musharraf, who is now under house arrest in the capital, rather than a plan to punish senior officers.
Sharif was at pains to absolve the military of blame for his overthrow at a recent news conference.
"I never had any trouble with the army. The coup was staged by one single person," he said.
"The rest of the army was not taken into confidence. We have to work together."
Officers say they are watching three things closely: the formation of a commission to investigate the 1999 invasion; who replaces the army chief in November when his term expires; and how Sharif plans to tackle militants, especially the Pakistan Taliban.
The army distinguishes between "good" Taliban - who do not attack Pakistani security forces and fight in Afghanistan - and "bad" Taliban, who have killed tens of thousands of soldiers and civilians in Pakistan.
Sharif favours talks with the Pakistan Taliban. Many in the military believe this could help address sokme individual grievances but they do not want any compromise on things like changing the Pakistani constitution.
The army wanted to know if Sharif could unite Pakistan's four provinces, each most likely headed by a different party, to rein in militancy, said Major General Mahmud Durrani, national security adviser under the previous government.
No slinking back to barracks
Pakistani militants have previously enjoyed the backing of the army, which used them as a proxy to attack India.
That has changed, officers insist, although elements within the military are still sometimes accused of supporting militant groups.
India blamed elements of the Pakistani security forces for supporting gunmen who attacked the Indian city of Mumbai in 2008, something Pakistan strongly denied.
Durrani said attitudes to India were slowly changing, meaning that top officers are unlikely to stand in the way of Sharif's outreach this time.
"For Pakistan to move ahead, it needs peace with India," he said. "But the military will not just slink back to (the) barracks. It still has a lot of input on foreign policy and needs to be weaned off that gradually."
The current army chief, Ashfaq Kayani, obliquely refers to India as an "external threat" in speeches but also now mentions the "internal threat" of militancy - a significant change in rhetoric.
"The army is now enlightened enough to understand the significance of peace with India," one high-ranking officer said.
"Everybody wants the Kashmir issue to be resolved."
But any changes would need to be slow and be matched with concessions from India, cautioned Durrani and Chaudhry, the former air vice marshal.
"It's a fallacy to imagine that foreign policy can be changed overnight," Chaudhry said.
"Can you wish away the problems with India? Can you wish away the problem of Kashmir?"
Sharif may also benefit from divisions between generals over the controversial extension of the term of the chief of army staff, said analyst Ayesha Siddiqua, who has written extensively about the Pakistani military.
Kayani has served two three-year terms, sowing dissent among some of those who had been in line to move up the ranks, she said.
Kayani's term is up in November, and Sharif will at least nominally have to approve his replacement. He had signalled he would accept the military's recommendation, she said.
"It seems like he's learned from his past mistakes," she said.
"He might use the Kargil commission (of inquiry) as a political tool to get some space to negotiate with India."
Sharif had three hours of talks with Kayani on Saturday.
"The army chief and Nawaz Sharif expressed their determination to fight terrorism," The News quoted a PML-N official as saying.
"Mr Sharif also praised the army's role in strengthening democracy and providing security during the elction."
The past five years had shown that while the military remained the most powerful force behind the scenes, it no longer wanted to take direct power, said retired Lieutenant General Talat Masood.
Musharraf, the last military dictator, was in detention. Generals had been hauled up before Pakistan's feisty courts and accused of vote-rigging, corruption and extrajudicial killings.
"The army does not have the monopoly on power it once did," Masood said.
"The performance of the last government was pathetic, but the army didn't step in - the judiciary did."