I can't confirm what you write for where I live.
Here the pools are crowded, in summer the lakes are crowded. You can't easily go to a swimming club just because you like swimming as a sport. In the clubs I know here, you have to show your talent and if you are not good enough, you can't join. And that's not because the clubs are self-opinionatet but because there are so many people who want to swim and join a club. They can't take all of them.
And Triathlon is very popular for years, there is not only running and cycling, but also swimming.
Perhaps it's also the question what you define as "swimming". For me the people who only go swimming for their pleasure, not for racing are also swimmers. Even if it takes them over an hour for a mile. Many of them are swimming nearly every day, so for me that's also sports even if they don't fight for speed.