There’s a specific issur of swimming with someone of the opposite gender that has nothing to do with how they’re dressed.
Also, I’ve heard b’sheim Rav Wolbe ZT”L not to close your eyes everywhere you go “If you don’t get the bekius, you won’t get the beiyun”. In other words, don’t try to look at people dressed inappropriately, but you also shouldn’t make a fuss about not looking at anything as it would just mean that your taivos will be that much greater when you accidentally see something.