Haven't seen you in awhile so we'll have to get together soon to go swim. As far as the critters are concerned, I wouldn't be too worried. In my experience seals, sea lions, and Harbor seals are curious but not overly excited to come over and say hello. That being said, when I've had those guys following me during a swim its more or less to figure out just what the heck I'm doing there
Best advice I can give is just keep swimming and they will more than likely leave you alone. In particular don;t make eye contact with a harbor seal (they ae grey with black spots and smaller than sea lions), they assume that by looking at them you want to play and they may come up and bump you or nudge your hand or foot. I'm use to it when they do this but it can be annoying when your trying to get a workout done.
Shoot me and emal and lets go swimming together soon!