South Africa's police commissioner Riah Phiyega announced that lead officer Hilton Botha was being replaced after local media revealed he faced seven attempted murder charges for having opened fire on a minibus in 2011.
"We recognise the significance, the importance and the severity of the matter," said Phiyega.
The country's top detective, Lieutenant General Vinesh Moonoo would take over the case, she said.
Charges against Botha, initially brought in 2009 and later withdrawn, had now been reinstated, said police spokesman Neville Malila. They themselves had only been informed on Wednesday, he added.
The development further embarrassed the prosecution, which has seen its evidence repeatedly picked apart during the bail hearing for Pistorius over the Valentine's Day killing of his girlfriend Reeva Steenkamp, 29.
Prosecutors do not accept the sprinter's explanation that he accidentally shot dead the model after mistaking her for a burglar. They have charged him with "premeditated murder" because of relationship problems.
But as the bail hearings enter a fourth day Friday, some lawyers following the case say the state has failed to present a strong enough case to keep Pistorius behind bars.
"He'll get bail because of the state's weak presentation of its case," attorney William Tintinger told AFP. "The state can't prove that the killing was premeditated."
Another attorney agreed, adding: "That's what everyone's been thinking who I've talked to."
Botha's removal from the case comes a day after Pistorius' defence team went after him in the witness box.
Summing up his case on Thursday, defence lawyer Barry Roux told the court: "The poor quality of the evidence of investigating officer Botha further exposed ... the disastrous shortcomings in the state's case."
Roux had earlier cast doubt on key prosecution witness evidence suggesting the couple, who had been dating since late last year, had had an argument before the shooting.
Prosecutors have also backtracked on allegations that police had found testosterone and needles in a dresser in Pistorius's bedroom. They later said the substance was unknown.
Botha himself was forced to admit that Pistorius's claims were "consistent" with the crime scene.
On Thursday he said of the investigation: "I'm sure it could have been handled better."
Steenkamp was found by medics in the early hours of Thursday last week at Pistorius's luxury Pretoria home covered in bloodied towels, with bullet wounds to her head, elbow and hip. She died at the scene.
In a statement read out in court earlier this week Pistorius said that he had fired at the door of the bathroom as he was "filled with horrible fear" that someone had sneaked in through an open window in the dead of night.
The 26-year-old athlete has previously said he kept a gun in his bedroom because of fears of violent burglary.
The Olympian and Paralympian sprinter, who has been in police custody for over a week, could face months or perhaps years in pre-trial detention if he does not win bail.
"There will be a level of shock in this country if he is not released," said Roux to murmurs of agreement from Pistorius family members sitting in the gallery.
But prosecutor Gerrie Nel on Thursday pointed out gaps in Pistorius' own account of the shooting.
"He fired four shots, not one. He meant to kill. On his own version, he's bound to be convicted," said Nel.
Under South African law someone who kills without immediate threat can be charged with murder.
Pistorius also could not explain how he ran past his bed twice without realising Steenkamp wasn't there.
"You want to protect her but you don't look at her?" Nel asked sarcastically, as the athlete huddled sobbing quietly.
Pistorius cut a sorry figure sitting alone in the dock, having lost weight and showing several grey hairs.
Magistrate Desmond Nair may decide to delay his finding until after the weekend.
Pistorius became the first double amputee to compete at an Olympics at the London games last year.
His family, ever present in court, said they would respect the bail process.
"This is a unique, emotional and difficult time for us all. Our hearts and thoughts are with the family of Reeva Steenkamp during their time of bereavement," said the athlete's uncle, Arnold Pistorius, in a statement.
Since the shooting the runner has lost endorsement contracts with Nike, sunglasses maker Oakley and French cosmetics firm Clarins.
Off the track he has had a rocky private life with stories of rash behaviour, beautiful women, guns and fast cars. He has built up a powerful team of lawyers, medical specialists and public relations experts for his defence.