Prime Minister David Cameron, condemning the "appalling" attack, added, "There are strong indications that it is a terrorist incident."
Eyewitnesses described how after the two men carried out their attack, they stayed at the scene, asking passers-by to photograph and film them.
The attack happened near the Royal Artillery Barracks in Woolwich, southeast London.
Armed police shot and wounded the two attackers after bystanders had called them to the scene.
Amateur footage of one of the men carrying a blood-stained knife and meat cleaver shows him saying: "We swear by almighty Allah we will never stop fighting you."
The man, a black man dressed in a hooded jacket and black woolly hat, speaks in a London accent.
"We must fight them as they fight us. An eye for an eye, a tooth for a tooth," he says.
"I apologise that women have had to witness this today, but in our land our women have to see the same. You people will never be safe. Remove your government, they don't care about you."
Reports said the man attacked was wearing a t-shirt bearing the logo of the British military charity Help for Heroes.
Several eyewitnesses said he had been decapitated.
Rapper Boya Dee, who witnessed the incident, wrote on his Twitter account: "Ohhhhh myyyy God!!!! I just see a man with his head chopped off right in front of my eyes!"
A woman who challenged the knife-wielding assailants said she intervened because "it was better having them (the weapons) aimed on one person".
Cub scout leader Ingrid Loyau-Kennett, 48, told the Daily Telegraph she had asked them why they had carried out the attack.
One of them told her they wanted "to start a war in London tonight", she told the paper.
The government's emergency response committee, Cobra, met hours after the attack and will convene again early on Thursday, Cameron revealed.
Security was reinforced at all army barracks in London following the incident.
Cameron, who returned early from talks with French President Francois Hollande in Paris, called the attack "truly shocking".
Hollande said at a press conference with Cameron that the victim was a soldier, but the British prime minister gave no confirmation.
Cameron said: "We've had this sort of attack before in our country and we never buckle in the face of them.
"People across Britain, people in every community, I believe, will utterly condemn this attack."
A local member of parliament, Nick Raynsford, said he believed the dead man was a soldier.
Home Secretary Theresa May said: "This is a sickening and barbaric attack."
The Muslim Council of Britain said the killers' use of "Islamic slogans" indicated they were motivated by their faith.
A statement from the council said: "This is a truly barbaric act that has no basis in Islam and we condemn this unreservedly. Our thoughts are with the victim and his family."
Police were called at 2:20pm (1320 GMT) to reports of one man being assaulted by two others.
Britain's top policeman confirmed two men had been arrested following the "shocking and horrific" attack.
"We have launched a murder investigation, being led by the Counter Terrorism Command," explained Metropolitan Police Commissioner Bernard Hogan-Howe
"Two men have been arrested in connection with that murder.
"It is hard to comprehend the shocking and horrific scenes we have seen this afternoon on a busy street as Londoners went about their day as normal."
Police commander Simon Letchford earlier said that "a number of weapons" had been reportedly used by the assailants.
Local officers, then firearms police officers arrived on the scene where they found a man who was later pronounced dead.
A white and blue police evidence tent was visible in the street and police tape sealed off the scene which was being scoured by officers in forensic suits.
Police later arrested two people as they tried to quell revenge attacks.
A 43-year-old man was detained in Braintree, southeast England, after reportedly walking into a mosque with a knife; another man was in custody in nearby Gillingham on suspicion of racially aggravated criminal damage to a mosque, spokesmen for the two forces confirmed.
Around 250 members of the anti-Islamist English Defence League were involved in minor scuffles with police at Woolwich Arsenal train station, near the scene of the attack, before dispersing.
A Buckingham Palace spokesman said Queen Elizabeth II -- who is due to visit the barracks later this month -- was "concerned" by the attack and was being kept updated.
The United States condemned the attack.
"We stand with our UK allies in the face of such senseless violence," State Department spokesman Patrick Ventrell said in a statement.
"Our thoughts and condolence are with the family of the victim and the British people."