You may need to make a mental identity shift before you can allow yourself to pursue this passion. Take this moment to realize that you have an inner circle that has shaped your values, and is continuing to do so.
This inner circle includes anyone whose approval you seek, whose judgment you value. For example, it might include your parents, your older brother, your middle school math teacher, your high school rival, your favorite professor in college, your college roommate, or even Earnest Shackleton (that Antarctic explorer who died years ago but you’ve read or at least heard of his biography). Whether you are aware of it or not, these people have shaped you. Think about what these people would say if you were to pursue your passion.
These people have been instrumental in shaping the person you are now. When you are contemplating a change, it may be time to temporarily distance yourself from their influence, because these influences are more likely to keep you where you are.
Real Life Example:
If you want to be a baker, go find some bakers to hang out with. Really get to know them and let their values rub off on you. They may let their values rub off on you. They may think your previous career is boring and dry but come alive when they talk about making a cheese souffle. They may describe it as the most worthwhile thing in the world to master.
A side benefit from this exercise is that you also get to have a sense of whether this new passion feels right to you by meeting people who are already passionate about it. Do you feel a sense of kinship with them? If you do, that’s further evidence that this might be a good fit.