Najib, 59, was already under pressure from conservatives in his ruling party for not delivering a stronger majority in Sunday's election despite a robust economy and a $2.6 billion deluge of social handouts to poor families.
Najib's Barisan Nasional won 133 seats in the 222-member parliament, well short of the two-thirds majority it lost in 2008. Opposition leader Anwar Ibrahim People's Alliance won 89 seats, up seven from the 2008 election but still well short of unseating one of the world's longest-serving governments.
"We could see Najib step down by the end of this year," said a senior official in the dominant United Malays National Organisation (UMNO), which leads the coalition.
"He may put up a fight, we don't know, but he has definitely performed worse. He does not have so much bargaining power," said the official, who spoke on condition of anonymity.
Former Prime Minister Mahathir Mohamad, still a powerful figure in UMNO, told Reuters last year that Najib must improve on the 140 seats won in 2008 or his position would be unstable.
Kuala Lumpur's stock market surged 7.8% on Monday to a record high on investor relief that the untested opposition had failed to take power. The Malaysian ringgit jumped to a 20-month high.
Ethnic Chinese who make up a quarter of Malaysians continued to desert Barisan Nasional, accelerating a trend seen in 2008.
Alarmingly for Najib, support from majority ethnic Malays also weakened, the UMNO source said, a sign that middle-class Malays are agitating for change.
"Malay unity is at stake here. Some of the Malays are rejecting UMNO. That is obvious," the source said.
ANWAR CRIES FOUL
Ethnic Chinese have turned to the opposition, attracted by its pledge to tackle corruption and end race-based policies favouring ethnic Malays in business, education and housing.
"We will work towards more moderate and accommodative policies for the country," a grim-faced Najib told a news conference after the majority was confirmed. "We have tried our best but other factors have happened ... We didn't get much support from the Chinese for our development plans."
Barisan Nasional also failed to win back the crucial industrial state of Selangor, near the capital Kuala Lumpur, which Najib had vowed to achieve.
Voting was marred by irregularities, said Anwar, 65, a former deputy prime minister in the 1990s who was sacked after falling out with his former boss, Mahathir. His three-party opposition alliance had been optimistic of a historic victory, buoyed by huge crowds at recent rallies.
But as counting went late into Sunday night, it became clear that his fractious opposition had failed to pull off what would have been the biggest election upset in Malaysia's history.
After claiming an improbable early win, Anwar later rejected the result as "fraudulent". He had accused the coalition of flying up to 40,000 "dubious" voters, including foreigners, across the country to vote in close races. The government says it was merely helping voters get to home towns to vote.
RISKS AHEAD FOR COALITION
Bank of America Merrill Lynch economists expect the government to focus on fiscal consolidation after a string of populist pre-election handouts. They expect cuts to fuel subsidies in the second half and a consumption tax next year.
It cited some risks, however, including UMNO elections in October and November when Najib may be challenged.
"Once the dust and excitement has settled, you will see that its not only the Chinese which were siphoned off from Barisan Nasional. There was a mini-Malay wave in the urban areas against UMNO," said another senior UMNO official.
"In the next round of elections within UMNO, you will see some dissidents emerging and asking for Najib to resign," said the official, who has held cabinet positions in government. He said Mahathir would be among those who back the dissidents.
The 2008 result signalled a breakdown in traditional politics as minority ethnic Chinese and ethnic Indians, as well as many majority Malays, rejected Barisan Nasional's brand of race-based patronage that has ensured stability but led to corruption and widening inequality.
Ethnic Chinese parties affiliated with Barisan Nasional suffered heavy losses in 2008 and were punished by voters again on Sunday. Its ethnic Chinese MCA party won just five seats, down from 15 in 2008.
That leaves Barisan Nasional dominated more than ever by ethnic Malays, who make up about 60 percent of the population, increasing a trend of racial polarisation in the country.
The benchmark FTSE Bursa Malaysia KLCI Index hit a lifetime high of 1,826.22, with stocks linked to the coalition and its favoured tycoons gaining handsomely.
Malaysia's second-largest lender by assets, CIMB Group Holdings Bhd, rose 8.8%. Its chief executive, Nazir Razak, is Najib's brother. Hospitals operator IHH Healthcare Bhd added 6.7% and energy services firm SapuraKencana Petroleum Bhd jumped 8.9%.
Australia's Lynas Corp Ltd, which is building the world's largest rare earths plant outside China in Malaysia, jumped 13%.
"There was a concern that the opposition would move fairly quickly against Lynas given that there were a number of groups actively protesting against the plant," said Michael McCarthy, chief market strategist at CMC Markets in Sydney.