it had launched the strikes after unmanned drones spotted suspected rebels of the Kurdistan Workers' Party (PKK), but said there were no civilians in the area and it was investigating the incident.
The attack, which Turkey's largest pro-Kurdish party called a "crime against humanity", sparked clashes between hundreds of stone-throwing protesters and police in Diyarbakir, the largest city in Turkey's restive mainly Kurdish southeast.
Police responded by firing water cannon and tear gas at the demonstrators. Seven people were detained. One police officer was hurt after being hit by a stone, witnesses said.
"We have 30 corpses, all of them are burned. The state knew that these people were smuggling in the region. This kind of incident is unacceptable. They were hit from the air," said Fehmi Yaman, mayor of Uludere in Sirnak province.
The Sirnak governor's office said 35 people had been killed and one wounded during an operation near the border with Uludere district.
The pro-Kurdish Peace and Democracy Party (BDP) said party leaders were heading for the area and that it would hold demonstrations in Istanbul and elsewhere to protest the deaths.
"This is a massacre," BDP Deputy Chairwoman Gultan Kisanak told a news conference in Diyarbakir.
"This country's warplanes bombed a group of 50 of its citizens to destroy them. This is a war crime and a crime against humanity," she said.
The Turkish military said it had learnt the PKK had sent many militants to the Sinat-Haftanin area, where the strikes occurred in northern Iraq, to retaliate after recent militant losses in clashes.
"It was established from unmanned aerial vehicle images that a group was within Iraq heading towards our border," it said.
"Given that the area in which the group was spotted is often used by terrorists and that it was moving towards our border at night, it was deemed necessary for our air force planes to attack and they struck the target at 2137-2224 (1937-2024 GMT)," it said.
"The place where the incident occurred is the Sinat-Haftanin area in northern Iraq where there is no civilian settlement and where the main camps of the separatist terrorist group are located," it said.
An investigation was in progress, it added, without referring to any deaths in the strikes.
The Turkish government, which has been battling the PKK since the group took up arms in 1984 to fight for an ethnic Kurdish homeland, was not immediately available for comment.
The incident threatens to spoil efforts to forge Turkish-Kurdish consensus for a planned new constitution that is expected to address the issue of Kurdish rights.
Corpses loaded onto donkeys
Smuggling is an important source of income for locals in provinces along the Iraqi border, with many villagers involved in bringing fuel, cigarettes and other goods from Iraqi villages on the other side of the border.
PKK militants also cross the border in these areas.
"There were rumours that the PKK would cross through this region. Images were recorded of a crowd crossing last night, hence an operation was carried out," a Turkish security official said.
"We could not have known whether these people were (PKK) group members or smugglers," he said.
Television images showed a line of corpses covered by blankets on a barren hillside, with a crowd of people gathered around, some with their head in their hands and crying.
Donkeys carried corpses down the hillside to be loaded into vehicles and taken to hospital.
Security sources said those killed were carrying canisters of diesel on mules and their bodies were found on the Iraqi side of the border.
They said the dead were from Uludere on the Turkish side of the border on what was a regular smuggling route.
The Firat news agency, which has close ties to the PKK, said that 17 people were still believed to be missing. It said those killed were aged around 17-20.
In northern Iraq, PKK spokesman Ahmet Deniz condemned the strike and said F-16 jets had bombed a group of around 50 people taking goods across the border and that 19 people were missing.
The PKK, regarded as a terrorist organisation by Turkey, the European Union and the United States, launches attacks on Turkish forces in southeastern Turkey from hideouts inside the remote Iraqi mountains.
Turkish leaders vowed revenge in October with air and ground strikes after the PKK killed 24 Turkish soldiers in one of the deadliest attacks since the PKK took up arms in 1984 in a conflict in which more than 40,000 people have been killed.