But, both the minister and the Congress party said he had been misquoted.
BJP immediately launched an attack.
"Any attempt to reduce the authority of the CAG or compromise the constitutional integrity of state institutions will be opposed inside and outside the Parliament," chief spokesperson Ravi Shankar Prasad said.
UPA's outside ally, the Samajwadi Party, echoed the sentiment.
"People in responsible positions should avoid making comments against constitutional bodies…We will not support such a proposal," party general secretary Mohan Singh said.
Senior CPI(M) leader Sitaram Yechury said it was not within the government's reach to take such a step.
"Any change in the CAG's composition requires a constitutional amendment…I don't think it is something that can really be ventured upon at this stage by the government."
On the other hand, information and broadcasting minister Manish Tiwari, while avoiding a direct comment on the issue, said the experiment of having a multi-member Election Commission has stood in "good stead over the decades".