This is going to sound stupid, but when I switched from regular swim trunks to jammers I was able to swim MUCH better. With trunks I felt (and still do) like I have umbrellas dragging behind me. LOL! That certainly added to the lack of progress down the lane, and subsequently, getting more easily winded.
It is natural to be able to breathe more easily on one side than the other, but keep practicing on both sides. It has taken me about 18 months to finally get more comfortable with bilateral breathing. Then you will be able to breathe when you need it. Breathing patterns are fine, but sometimes you must alter them. Maybe you are sharing a lane and someone swims by splashing you in the face just as you are breathing.
So yes, practice, practice. And keep aware of how high (or low) your hips are in the water. Higher hips make it easier to breathe.