He said that Singh had also accepted that he had recommended for a passport for Khan in 1997.
"It is thus inappropriate to retain him in the post of Lt Governor," he said.
Earlier, Loku Ayyappan and delegates of other outfits tried to meet media persons to air their plea but were prevented from doing so, following which some of them staged a demonstration near Raj Nivas and were arrested.
Police said all those arrested were subsequently let off.
The Prime Minister's Office has given its nod to the Enforcement Directorate to question Puducherry Lt Governor Iqbal Singh this week.