Obama also offered his most vigorous public defense yet of drone strikes as legal, effective and necessary.
"Neither I, nor any president, can promise the total defeat of terror," Obama said in a speech at the National Defense University. "What we can do - what we must do - is dismantle networks that pose a direct danger, and make it less likely for new groups to gain a foothold, all while maintaining the freedoms and ideals that we defend."
Obama also implored Congress to close the much-criticized Guantanamo Bay detention center in Cuba and pledged to allow greater oversight of the controversial unmanned drone program. But he plans to keep the most lethal efforts with the unmanned aircraft under the CIA's control.
Obama's address came amid increased pressure from Congress on both issues. A rare coalition of bipartisan lawmakers has pressed for more openness and more oversight of the highly secretive targeted drone strikes. Liberal lawmakers have pointed to a hunger strike of more than 100 prisoners at Guantanamo – the military earlier this month was force-feeding 32 of them – in pressing for stalled efforts to close the detention center to be renewed.
The president cast the drone program as crucial in a counterterror effort that will rely less on the widespread deployment of US troops as the war in Afghanistan winds down. He said he is deeply troubled by the civilians unintentionally killed.
"For me, and those in my chain of command, these deaths will haunt us as long as we live," he said. Before any strike, he said, "there must be near-certainty that no civilians will be killed or injured - the highest standard we can set."
In Pakistan alone, up to 3,336 people have been killed by drones since 2003, according to the New America Foundation, which maintains a database of the strikes. However, the secrecy surrounding the program makes it impossible for the public to know for sure how many people have been killed and how many were intended targets.
The Justice Department revealed for the first time on Wednesday that four US citizens had been killed in US drone strikes abroad. Just one was an intended target - Anwar al-Awlaki, who officials say had ties to at least three attacks planned or carried out on US soil. The other three Americans, including al-Awlaki's 16-year-old son, were unintended victims.
Some Republicans criticized Obama as underestimating the strength of al-Qaida and for proposing to repeal the president's broad authorization to use military force against the nation's enemies - powers granted to George W Bush after the Sept. 11, 2001, attacks.
"I believe we are still in a long, drawn-out conflict with al-Qaida," Sen. John McCain, a top opposition Republican, told reporters. "To somehow argue that al-Qaida is on the run comes from a degree of unreality that to me is really incredible. Al-Qaida is expanding all over the Middle East, from Mali to Yemen and all places in between."
Obama announced new "presidential policy guidelines" on the standards his administration uses when deciding to launch drone strikes. According to an unclassified summary, the US will not strike if a target can be captured, either by the U.S. or a foreign government; a strike can be launched only against a target posing an "imminent" threat, and the US has a preference for military control of the drone program.
The CIA will continue to work with the military on the program in Yemen and control the program in Pakistan, given the concern that al-Qaida may return in greater numbers as US troops leave Afghanistan.
The guidelines will apply to strikes against both foreigners and US citizens abroad.
Drone strikes will largely remain highly secret for the public. Congress has been briefed on every strike that drones have made outside Afghanistan and Iraq, Obama said, but those briefings are largely classified.
While civil rights groups welcomed some of Obama's steps, they appeared largely unappeased.
"The president still claims broad authority to carry out target killings far from any battlefield, and there is still insufficient transparency," said Anthony D Romero, executive director of the American Civil Liberties Union.
Obama was interrupted three times by a woman from the anti-war group Code Pink, who appeared to be protesting both the drone program and the Guantanamo prison. The president acknowledged that the issues were worth being passionate about.
In seeking to close Guantanamo, many Republican lawmakers oppose Obama's efforts to bring some of the detainees to the US to face trial and be held in maximum-security American prisons.
But the new hunger strike by prisoners protesting their conditions and indefinite confinement has refocused the president on efforts to close the detention center. He announced a fresh push to transfer approved detainees to their home countries and lift a ban on transfers to Yemen.
Thirty of the 56 prisoners eligible for transfer are from Yeman. Obama halted all transfers to the poor Middle Eastern nation in 2010 after a man trained there was convicted in a failed bombing attempt of an airliner over Detroit.
In a statement, the Yemeni government said it welcomed the administration's decision and pledged to "work with the United States to take all necessary steps to ensure the safe return of its detainees."
The president said he planned to appoint a special envoy to oversee the prisoner transfers and other efforts to close the prison.