When I'm triathlon training I tend to shift focuses between disciplines fairly regularly. But I never drop any of the non focus disciplines completely. There's always something to do with them.
Cross training beyond the 3 disciplines for me involves weekly strength training using a TRX (primarily) and some free weights. Plus I have a single Pilates Reformer session per week. I'm mostly trying to build core strength rather than pure power. I find the focus on core helps tremendously and also is vital to my pain management due to a chronic backs issue.
As for swimming: I find that I use the non-swim weeks (where my swimming yardage is reduced) a great time to focus on form and technique rather than base building or other such items. I'm also swimming briefly after every long run or hard run/bike ride. 10 -15 minutes in open water or a pool just stretching out and doing something other than hammering the various leg muscles. I do not count those sessions in my weekly training totals btw, but they're great for form work. I find these shorter sessions far more valuable than x reps of lat pull downs in my strength training (though those help as well).