Talvi, I went for a walk and was trying to imitate your 'swimming breathing' and only got breathless when I held my breath regularly for a second or two. I wonder therefore if you are doing what I did, up to a few weeks ago, and you are holding your breath at some stage of the cycle or breathing out in a staccato fashion? If you think about it, to your sub-conscious, laying face down in the water and not breathing even for a short time is not normal and must create anxiety, especially when you don't respond immediately and get yourself out of that situation.
So try swimming slowly and start breathing out, in a slow steady stream, the moment your face returns to the water, no holding your breath. Just like you breathe on dry land in-out no pause a steady flow. Not original I know, the DVD and the coaches all say breathe out slowly and steadily underwater, trickle breathing, but easily overlooked, and I got in the habit of breathing out staccato fashion with each arm stroke.
Let us know if it helps, it worked for me, and distance is building.