"Given my confidence and the multitude of new commitments I am taking on now, I think it would be impractical for me to continue to serve on the board, and it is best for me to move off," Ballmer said in a publicly posted letter to Nadella.
"I bleed Microsoft - have for 34 years and I always will."
Nadella, in a responding letter posted online, thanked Ballmer for his contributions to Microsoft and said he supported the decision to leave the board.
Ballmer said he would be devoting time to new endeavors that include teaching and the Los Angeles Clippers NBA team he acquired earlier this month.
Ballmer paid a record $2 billion for the Clippers.
Microsoft last month announced its biggest job cuts ever as new chief executive Nadella called for a new focus at the US tech giant while integrating the Nokia phone division.
The company said it would slash 18,000 jobs from its global workforce over the next year, the majority from the Nokia handset unit acquired this year.
Nadella, who became CEO earlier this year, seeks to reinvigorate a company that had been the world's largest but which has lagged in recent years as Google and Apple have taken leadership of the tech sector.