You can think of regular breathing as being like an aborted roll to your sweet spot.
First, make sure you are exhaling underwater before you roll to breathe. When you're rolling to your sweet spot, there's time to both exhale and inhale, but in regular breathing there isn't time to do both, and you won't be able to inhale if your lungs are still full of air.
Second, make sure that you are using your body roll to bring your nose and mouth to the air, regardless of whether you are going to your sweet spot or doing regular breathing. If you are doing triple-switch drills, your nose should stay pointed at the bottom during the first two switches, but your head should roll with your body during the third switch, as though someone has superglued a connecting rod between your shoulder and your chin. (When you're doing the triple-switch drills, you can remember this by thinking "one, two, superglue.")
Third, start experimenting with how far you actually need to roll to get air. Terry did an article on breathing for Total Swim
back in 2005:
and I think the pictures in it are as important as what he said. Notice, in particular, how Terry is able to breathe without even rotating all the way onto his side, let alone to his sweet spot. Roll only far enough to quickly fill your lungs, then recover your arm and point your nose back at the bottom.