I don't how helpful or relevant this might be for you, but I've found that it's important to put my face back in the water immediately after quickly getting a breath on the breathing stroke. I had gotten in the habit of waiting for the recovering arm to pass my head on the way to the spear, and then turning my face back into the water along with the arm.
Now that I'm paying attention to putting my face back in the water immediately, I have felt like my torso is a much more stable platform, completely unrelated to the motion of turning my head to breathe, and my stroke is more symmetric. This makes my weak side breathing feel as good as my strong side breathing--I'm just more symmetrical and stable overall. Might be worth exploring for you to see if the same applies in your stroke.