How frequently are you breathing? How much pause-and-glide are you putting in your strokes? I find too slow a stroke rate and my breathing isn't adequate enough to be comfortable, even breathing every 2nd stroke. Try shortening your stroke (sacrilege I know) and take more strokes to help with your breathing rhythm. You can always lengthen it out and slow the rate down as you get more metabolic efficiency. If you feel like missing one breath would be disastrous and have your body screaming for air you should definitely breath more frequently.
If you breath every 3 currently you could try every 2 to the left one lap, then to the right on the next lap or another pattern like 2-2-3-2-2-3... or every 3 in the middle of the lap, every 2 in and out the wall.