Firefighters battled through the night to douse the flames at the Aswad Knit Composite factory at Sripur, on the outskirts of the capital Dhaka. Parts of the two-storey building were still smouldering early on Wednesday.
Police said the fire, which broke out when most of the 3,000 workers had left, was so intense that most of the bodies were too badly burned to be identified.
"Two bodies have been identified and handed over to their relatives. Five other bodies were charred beyond recognition," local police chief Amir Hossain told AFP, revising down an earlier toll of nine.
Workers said the overnight blaze appeared to have been started by a malfunctioning knitting machine which had caught fire on previous occasions.
An AFP correspondent at the scene found work order books containing names of clients in September which included US brand Gap, British retailer Next, Swedish fashion label H&M, Australia's Target and French supermarket Carrefour.
Both Gap and H&M denied that they had placed orders for clothes at the factory, but said they had a supplier relationship with its owner Palmal Industries, one of the country's largest garment groups.
"This factory does not produce garments for H&M, hence we don't audit this factory," H&M spokeswoman Andrea Roos told AFP by email.
When asked to comment on the order marked for the company, she said the factory was a supplier of fabric and yarn but "we don't have a direct business relationship with the affected factory."
A local Gap executive, who asked not to be named, also denied the group's clothes were made there.
A fire at the Tazreen garment factory in Dhaka killed 111 workers last November, the country's worst such tragedy, and revealed unauthorised subcontracting of orders from Western groups including Walmart.
In May Stockholm-based H&M conceded that orders had been placed without its knowledge at a factory in Cambodia where 23 workers were injured when a metal and concrete structure collapsed.
Police chief Hossain said the fire was now under control although parts of the building were still smouldering. Firefighters complained overnight about a lack of water in the area.
Industrial accidents are common in the country, where the collapse of the infamous Rana Plaza factory complex in April killed 1,129 people. The disaster focused attention like never before on conditions in the industry.
Safety standards at Bangladesh's 4,500 garment factories, where workers toil for 10-12 hours a day for a monthly minimum wage of $38, are notoriously lax and fires are a common problem.
The government promised a clean up after the Rana Plaza tragedy. Ninety leading brands, mostly from Europe, signed up to a new safety agreement drafted by trade unions to allow greater scrutiny of their operations.
Major US retailers including Walmart refused to join in.
The country has the world's second biggest clothing industry, worth $21.5 billion a year. It produces 80 percent of Bangladesh's exports, most of which are destined for Europe and the United States.
Thousands of garment workers walked off the job last month, blocking roads and attacking factories outside the capital to demand a $100 minimum monthly wage.
Pope Francis has said that workers are paid so badly that it is akin to "slave labour".
"It's quite clear that safety issues were ignored (at the Aswad factory)," Kalpona Akter, head of the Bangladesh Center for Worker Solidarity, told AFP.
"This fire is yet another red alert for the Western retailers who still have not done anything for the safety of Bangladesh's millions of garment workers despite a series of fire and building disasters in recent months," Akter added.
Although the names of the victims in the latest fire have yet to be released, relatives who had gathered at the factory feared the worst.
Sumi Akter, 22, said she suspected that her husband Bulbul Islam was among the dead.
"He called me by phone last night and said he would be back home in an hour. But after the fire I heard nothing from him," she told AFP.
Most workers had left before the fire started.
Mohammad Abu Saan, who works at the factory in Sripur, said the fire started when a knitting machine burst into flames on the factory floor and the blaze spread to a warehouse.
"There have been quite a few small fires in the machine recently. But we managed to douse them. This time it was big," he said.
Earlier, police and fire officials said the blaze started in a boiler.