I swam through a midwest, U.S. winter last year in an indoor pool and had no problems. But a number of years ago I swam for several months in a well-kept university outdoor pool, because of a running injury, and I got 3 "colds" in 3 months. As it turned out, the illnesses were likely sinus infections (bacterial) rather than typical colds (viral). The cooler water apparently gummed up the flow of my sinuses and allowed the overgrowth of bacteria, giving me cold-like symptoms. Antibiotics were required to clear up all three infections.
After the third bout, I crawled out of the pool and resumed running and had no further issues.