Abe has been criticised by the opposition for playing several rounds of golf on the day of the disaster before breaking it off to rush back to Tokyo. Critics have also slammed his returning to his vacation villa afterwards.
Among those killed were two brothers, aged 11 and 2, and a firefighter who was engulfed by mud as he was carrying a toddler to safety. The child also perished.
"There was a really strange smell, a very raw, earthy smell. When we opened a window to see what was going on, the entire hillside just came down, with a crackling noise, a thundering noise," one woman told Fuji television.
She and her husband fled just before their house filled with mud, leaving 5 metre boulders where they had been sleeping. About 240 mm (9 inches) of rain fell in the area about 3 km from the centre of the city in the 24 hours up to Wednesday morning, the Meterological Agency said. Roughly half of that rain fell in one hour.
Cities in land-scarce Japan often expand into mountainous areas, with houses tucked just below steep slopes, leaving them vulnerable to landslides.