As Stuart said, it could be just be a matter of degree, and you may be getting ahead of yourself on the exhale. Most of us can stand or sit on the bottom if we exhale enough.
On the other hand, there are sinkers. In the sixties, I taught swimming at a scout camp on Long Island. Of the what must have been thousands of swimmers I came across, there was one true sinker. He could fill his lungs, go down to the bottom, and do push-ups on the sand. Despite that, he did not struggle in the water. He was a good swimmer, was actually one of our fellow waterfront counselors.
Back to your question. When I first tried them a couple of years ago, I had and still have to some extent, trouble breathing on those skate and/or spear drills. I'm sure I am not the only one. If you look at some of the TI videos, Easy Freestyle or O2 in H2O, (not sure which, maybe both) you may notice that in some of the demonstrations, Terry is almost on his back, his face to the sky. (Dolphins and whales get to breathe with their noses pointing straight up.)
In these demos, he is almost stationary in the water, making very little headway. In other demos, with the dynamics of forward motion and angular momentum to help (it is beyond me right now to explain just how) you see him not rolling nearly so far to breathe.
A good early sequence for you might be to roll up far ehough that you can take a couple of comfortable breaths, roll back down and do your switching and gliding. When you want air again, roll up on the other side or switch again and roll back up on the first side, and take a couple of breaths. You will likely find it works better on one side than the other.
Don't spend too much time on this all at once, but do come back to it again and again. It will get easier. You will find yourself not having to roll up as far and spending more time looking down at that black stripe.
Forward motion does help. Swim fins are often mentioned at this stage. If you are not getting enough drive out of your kick, they may be just the little extra you need to get this working.