The unmanned submarine loaded with sonar deployed Monday night from the Australian ship Ocean Shield which has spearheaded the hunt for the Boeing 777 that vanished on March 8.
"After completing around six hours of its mission, Bluefin-21 exceeded its operating depth limit of 4,500 metres (15,000 feet) and its built in safety feature returned it to the surface," Australia's Joint Agency Coordination Centre said.
"The six hours of data gathered by the Autonomous Underwater Vehicle is currently being extracted and analysed," JACC said.
The AUV had been due to spend 16 hours collecting data.
US Navy Captain Mark Matthews said the vehicle had exceeded programmed operational limits and automatically resurfaced.
"In this case the vehicle's programmed to fly 30 metres over the floor of the ocean to get a good mapping of what's beneath," he told CNN from Perth.
Charts had put the depth at 4,200-4,400 metres, he said.
"It went to 4,500 metres and once it hit that max depth, it said this is deeper than I'm programmed to be, so it aborted the mission."
Search zone adjusted
Matthews, a search and recovery expert, said the crew would now refine the task to cope with the depth encountered.
"It happened in the very far corner of the area it's searching. So they are just shifting the search box a little bit away from that deep water."
The US navy's AUV Bluefin-21 would embark on a second mission during the day, weather permitting, JACC said.
Agency chief Angus Houston announced Monday the end of more than three weeks of listening for signals from the plane's black boxes and the launch of submarine operations.
Phoenix International workers Chris Minor (R) and Curt Newport inspect the US Navy’s Bluefin 21 autonomous underwater vehicle (AUV) before deployment in the southern Indian Ocean to look for the missing Malaysia Airlines flight MH370 in this handout picture released by the Australian Defence Force. (Reuters)
The vehicle would survey the silty ocean floor for 16 hours at a time to gather a maximum amount of data, he said.
"Bluefin-21 is planned to redeploy later today when weather conditions permit," JACC added Tuesday.
The forecast for the search zone 2,170 kilometres (1,345 miles) northwest of Perth is southeasterly winds with scattered showers and isolated thunderstorms, sea swells up to two metres (nearly seven feet) and visibility of five kilometres (three miles).
Oil slick found
Retired air chief marshal Houston also revealed Monday that an oil slick had been sighted in the search area, with two litres collected for testing.
Malaysia Airlines Flight MH370 went missing en route from Kuala Lumpur to Beijing with 239 people on board. The cause of the disappearance remains a mystery with no debris found despite an enormous search, involving ships and planes from several nations.
It would take several days to test the oil ashore, but Houston said he did not think it was from one of the many ships involved in the hunt.
It is 39 days since the plane vanished and the batteries powering the black box tracker beacons had a life of only 30 days.
Ocean Shield detected four signals linked to aircraft black boxes, but the last confirmed ping came on Tuesday last week and officials suspect the batteries are now dead.
Houston stressed the enormous difficulties of working at great depths in such a remote location.
"This is an area that is new to man," he said, adding that while it was rolling seabed rather than mountainous, a lot of silt was on the bottom.
"And if we have silt on the bottom, that can be quite layered, quite deep and that will complicate how things are on the bottom."
The Bluefin-21 is equipped with side-scanning sonar and will initially focus on 40 square kilometres (15 square miles) of seabed in the vicinity of the detected signals.
But Houston explained that the vessel operates slowly, with each mission taking 24 hours to complete.
The 4.93-metre long Bluefin device, which weighs 750 kilograms, needs two hours to reach the bottom, where it produces a high-resolution 3D map before surfacing in another two hours.
In this photo provided by the Australian Defense Force an autonomous underwater vehicle is deployed from ADV Ocean Shield in the search of the missing Malaysia Airlines Flight MH370 in the southern Indian Ocean. (AP Photo)
Downloading and analysing data requires a further four hours.
But while the mini-sub could take the search a step closer towards visually identifying any wreckage, Houston repeated his longstanding note of caution.
He recalled that after Air France Flight AF447 crashed in the Atlantic Ocean in June 2009, it took nearly two years to retrieve the black boxes from a depth of 3,900 metres.
"However, this is the best lead we have and it must be pursued vigorously. Again I emphasise that this will be a slow and painstaking process."
See full coverage