"How dare you say this?" Modi said, raising his voice. "What right you have to say this?"
Modi said that while he was willing to take on Manmohan Singh within the country, he would not tolerate any attack on the Indian prime minister by a foreigner.
"There can never be a greater insult to the Indian PM. There cannot be a greater insult to India," he said of Sharif's reported remark.
And as the crowds roared, Modi added: "We will not tolerate any insult to India."
Modi's comments came hours before Manmohan Singh and Sharif are set to meet on the sidelines of the UN General Assembly.
The BJP leader said Sharif's description of Manmohan Singh as a "dehati aurat" had caused him great grief.
He added that the reason Sharif "insulted" Manmohan Singh was because the Indian prime minister was not being respected by his own Congress party.