I agree with Techie. Next time it happens just throw on your biggest cheshire cat grin and remember this exchange from Alice in wonderland
One day Alice came to a fork in the road and saw a Cheshire cat in a tree. "Which road do I take?" she asked.
"Where do you want to go?" was his response.
"I don't know," Alice answered.
"Then," said the cat, "it doesn't matter.”
I've started attending a masters class on occasion at the pool where I swim and get similar comments about my kicking, or perceived lack thereof. I sometimes question my decision to "limit" my kick because unlike the majority of other swimmers there I wear a size 15 shoe and should (theoretically) get some good propulsion out of it. Would I like to get more out of my natural flippers and go faster? Sure (I also dream about having webbed fingers and toes every now and then) but not at the expense of my easy stroke and 2-beat kick which carried me through my first open water 10k swim over the weekend just fine. I will occasionally use a 6 beat kick at masters class just to make some bubbles behind me so they'll keep quiet and I can focus on my stroke, but I never employ the spasmodic effort I think they are expecting. I did explain to one of the coaches (former college sprinter, I think) that my goal is to complete and compete in more open water distance swims, not just faster laps in the pool and he seemed to understand.
If your goal is whitewater, and short bursts of looking fast then by all means, kick like an idiot. If your goal is a speed that you can maintain over a distance of your choosing then you need to be aware of where you're expending and/or wasting energy and how to maximize arm, leg and total body movement on each and every stroke. Don't worry about the well-meaning comments of others who are "better" swimmers than you. A coach probably yelled at them for several hours a day until they did it his/her way. There are many different methods/theories out there on how to swim the best, and they keep evolving. You're learning to swim using a method that you've chosen; a method, in my humble opinion, that helps you understand what your doing and why your doing it better than most.