Could be the wind was creating a current. Or perhaps your bodies weren't as fluid and relaxed as when your muscles were warmer. The water molecules would be a little closer together when cold, but 3 or 4 degrees wouldn't make a huge difference in that.
On weekends I try to do a long session in a 50 meter chlorine pool. Compared to the salt water pools I normally swim in, the water feels "thin", and not in a good way. I want it to feel a little thicker when I'm making my catch.