You are not alone!
To try and shorten the time for you to feel under control with your breath, I would suggest a focus on breath control for the next 6-12 sessions during your warm up.
How much air do I inhale?
How much air do I exhale?
Do I empty my air chamber? or keep some air in the tank?
Does the amount make a difference to my breath control?
Is my breath relaxed? Is it rushed?
Do I use my nose and mouth to inhale and exhale?
Does it make a difference if I start out breathing bilateral? or every 2 strokes?
Find out everything you can about your breath and with trial and error, you will find the best oxygen control strategies for swim.