The whole dimension of spirituality is to grow beyond the physical. If your involvement with the physical is very deep then naturally your attachment to the body is very strong. It is not sex per se which can be an impediment in one's growth but generally sexuality breeds an attachment to the body; in that context this has been said.
But does the physical act itself stop a person from pursuing his spiritual path? Not at all, but what is being said is, unless your focus is strong and pointed, the chances of progress or the progress itself will naturally be very slow because you are thinking in many directions at the same time. For a person who wants to go all out and do things in a short time, the advice is definitely not to get entangled with these bodily aspects.
Though people would like to project their sexuality as a very conscious process, it is not so. It is a compulsive process. Maybe, it is handled a little consciously by some people while others do it compulsively but essentially the ingredient, the basic seed of sexuality is a compulsive process. It is physiological and chemical; it is the function of the hormones which is driving you like that.
So, unless you are able to transcend all compulsiveness within you, definitely it is an impediment in that sense. Any kind of activity, whether it is eating or sexuality or talking or whatever, if it is a compulsive process, if you are enslaved to it, then that is an impediment.
If you look at what is causing this compulsiveness, if you understand that and handle that aspect, then compulsiveness goes away. That's definitely moving towards freedom.
If one pursues a spiritual path, irrespective of where he is right now, gradually he may evolve out of many compulsions within himself.