Nearly 8,600 people were taken to the emergency room for heat-related symptoms in the week through Sunday, more than double the previous week, the Fire and Disaster Management Agency said.
The elderly were particularly vulnerable to the rising temperatures, with nearly half of those taken to hospital aged 65 or older, the agency data showed.
The Japan Meteorological Agency has warned that the mercury was expected to top 35 degrees Celsius again on Tuesday.
Last year, Japan experienced its hottest summer on record, with the mercury hitting a record 41 degrees C (105.8 Fahrenheit) in some parts of the country.
Tens of thousands were taken to hospital suffering from heat-related symptoms.
The archipelago nation regularly experiences sweltering summer temperatures.