I'm not sure I agree about the increase in appetite. I don't feel any hungrier after swimming outdoors in cold lake or sea water than I do after swimming in an indoor heated pool. However, I have gained weight on Swimtrek tours, even while consciously trying to eat no more than usual.
An Irish couple on the last trip were describing their local open water club, and mentioned a clubmate who had a thick layer of fat all around his torso from swimming every day, year 'round, in the Irish sea with no wetsuit. He referred to it as his "bioprene".
When Lynn Cox was training for her extreme cold water swims, she was carefully monitored by hypothermia researchers. She added body fat even with no extra calories. So perhaps the body tries to protect itself by adding the insulation.
In any case, it's subcutaneous fat, which has no ill effect on health. It's the viscous fat - the classic "beer belly" - that does the damage over time.