I have participated in quite a few sports (poorly, even my best ones) and I have done what I think are nearly most types of training sessions (high effort, medium performance), but it is always refreshingly important to remind yourself of what they all feel like. Sure, glycogen depletion - and I mean proper depletion, is the same/similar however you get there; all-out exercise of 30 to 200 seconds is just plain horrible; getting hot can really hurt your soul; all in the name of training ->adaptation-> progress. So once you have covered exhaustion and fatigue a number of different ways, for some, it can be almost over familiar.
But when it comes to prescribing sessions, it is crucial to a sports physiologist's skill set to know what that session feels like, to know when the pain starts to kick in, to know how hard the breathing gets toward the end, to know how pain can alter the space time continuum by slowing the passing of time.
I am convinced that you do not necessarily need to have been an elite athlete to coach or support elites (I would say that though wouldn't I?) but I am sure that you need to have experienced the rigors of an unremitting training programme. You need to have gone 'through the mill', felt the horrors of a repetition set, felt dizzy from full exhaustion, felt the cramps of depletion. Just as importantly, you need to have critiqued the sessions you have gone through, using your scientific knowledge of what is going on inside. In my view, you cannot be a sports physiologist of full Jedi master status unless you have been through this cycle of learning, doing, critiquing, re-learning.
I am also sure there is no better tool to engage with real athletes other than being able to describe the feelings and subtle sensations that occur during training, for example, "You know what your breathing feels like after 400m of an 800m race", is as effective a way of introducing the need for inspiratory muscle training as any. I think, immediately the athlete can appreciate that you understand their world, rather than you launching into a technical splurge just because you feel you should and your training says you should. To put it another way, I think an athlete can smell the absence of session empathy - that could also be from a lack of understaning of training units, minute/miles, power outputs, recovery time. A good physiologist has read about these sessions in the literature, studied them in the textbooks and then had a go at them. There is no substitute for that real experience.
So next time you put pen to paper, as author of a training programme, go and do the session yourself first. But crucially question it, what it is doing and why it is important. Try before you apply!