Welcome to the forum! And Excellent post.
This is a common problem and I see it frequently. You are emptying your lungs too quickly too early causing you to sink. Many triathletes are told to immediatley empty lungs when face hits the water. You do want to exhale (slowly), but not empty too fast. Empty lungs when rolling to get breath and refill tank quickly.
Each swimmer needs to experirment with their body type on how much air in lungs is needed to maintain buoyancy. A good test is (in pool) is fill lungs (big deep breath) then hold knees to chest and bob face down at surface for a moment. Very slowly exhale until you start to sink. For me, just under 50% capacity, the ship's going down.
Here's a SwimVice video on this very subject that may be helpful too: SwimVice: Breathing and Buoyancy