I live in Sweden and my house is right at the lake side. In the spring I start to jump in at about 12-13 C and it is terrible, really ice cold. I hate it at that time of the year. During the summer the lake warms up to about 23-24 C at the peak. Now we are at 17 and in September we will gradually fall down to 12-13 C again, but in the fall I love it. I feel great in the cold water. So the conclusion is, you need to adjust. To swim in 16 degree water when you are used to 32, no wonder you don't like that!
It looks like Naj has the best experience and I think you should listen carefully to his advice.