A Reuters photographer saw him being escorted by police through a crowd of about 100 angry demonstrators, some waving Russian flags, near the headquarters of a foreign observer mission.
Crimea is under control of Russian forces who seized it last week, although Moscow says "self defence" units of men in uniform without insignia are not under its command.
"UN special envoy Robert Serry with me in coffee shop. Outside local militia block the door," Mates tweeted.
"He refused to go with men blocking car, got out and walked until he found coffee shop. He's asked ITV News team to stay with him," Mates said. Later Mates added that Serry had said he was very happy to leave Crimea "if it helped de-escalate the situation".