agency chief Steven Miller and promised a new system of checks and safeguards to make sure the episode was not repeated.
"Given the controversy surrounding this audit, it's important to institute new leadership that can help restore confidence going forward," Obama said.
The president, who has dismissed Republican attempts to link him to the scandal at the independent IRS agency, also pledged to work directly with Congress as it carries out its oversight duties in the matter.
Obama said that evidence of abuses in the tax agency, revealed by a report by a government watchdog released on Tuesday, were "inexcusable."
"Americans have a right to be angry about it, and I am angry about it. Obama said at the White House, after meeting top Treasury Department officials.
"I will not tolerate this kind of behavior in any agency, but especially in the IRS," he said.
Scandal erupted when it emerged that officials studying tax exempt status applications singled out groups with names including phrases like "Tea Party" or "Patriots," which could thus be expected to be fiercely opposed to Obama.
Republican congressman Darrell Issa told CNN that Obama had taken a "good first step" and promised a robust congressional probe into the scandal.
"The president will find very willing partners on Capitol Hill," he said.
"I think in this case, we very much take him at his word that he wants to be open and transparent, in fixing the system and putting new controls in place."
But top Senate Republican Mitch McConnell signaled that the controversy was far from over as his party tries to use it to slow Obama's political momentum at the beginning of his second term.
"These allegations are serious -- that there was an effort to bring the power of the federal government to bear on those the administration disagreed with, in the middle of a heated national election," McConnell said.
"We are determined to get answers, and to ensure that this type of intimidation never happens again at the IRS or any other agency."
Earlier, Attorney General Eric Holder meanwhile promised a nationwide probe into the claims as he endured a grilling from angry Republican lawmakers on the IRS flap at a hearing in the House of Representatives.
"The facts will take us wherever they take us," Holder said, promising the probe would spread nationwide, beyond the Ohio city of Cincinnati, where the violations are alleged to have taken place.
Holder said that he did not know whether the alleged misbehavior at the IRS was a low-level affair or if more senior officials were implicated.
He argued that some examination of the tax-exempt status of independent groups was probably necessary -- but should not "depend on the political persuasion of the group."
Republicans meanwhile sought to spin the IRS scandal, and other recent controversies currently hampering the administration, into a wider narrative, suggesting systematic abuse of power within the administration.
"My question isn't about who is going to resign. My question is: 'Who's going to jail over this scandal?'" House Speaker John Boehner said.
"Clearly someone violated the law."
Republican Senator Marco Rubio said the IRS drama was symptomatic of a White House obsessed with political campaigning.
"This is a culture of intimidation, a willingness to play hardball politics against your political opponents," said Rubio, a potential 2016 presidential candidate.
"Few times in the history of this country has anyone used this office to drive more wedges among the American people than this president and this administration."
Congressman Charles Boustany, who has looked into allegations of IRS abuse for two years, said that "after months of misleading testimony and evasive answers... they've now been caught red-handed."
"I can tell you right now this culture of rot and abuse at the IRS is going to be rooted out."
The watchdog report by a Treasury Department inspector general published on Tuesday found that some Internal Revenue Service staff singled out grass roots groups opposed to Obama.
The White House says no one in Obama's governing or political team had anything to do with targeting the groups over their tax-free status, and Obama has said he only learned about it from news reports last week.