I started yoga in my 20s, but I feel my attitude to yoga has developed and improved as I have entered my 30s.
Earlier, a lot of my practice was fueled by ego. I had to go deeper into the posture than others in the class. My shoulder stand had to be the best.
Slowly, though my flexibility has gone down somewhat, my yoga practice has improved. I pay more attention to the details - my breath, how a minute adjustment alters my entire posture, how each day my body reacts differently to the same posture.
The aim of yoga is to still the mind, not contort your body into gymnastic postures. To this end, it doesn't matter at all what age you start yoga. If anything, you may find it easier to focus on the true spirit of yoga as you grow older.