There are lots of things tax payers pay for that you cannot use. Should people pay for a Library if they dont go, Should people without Kids pay for schools?
Should we pay for EMS when we use Hatzolah?
Should people who dont drive pay for roads.
That is the price we pay for living in a free society, we have to pay for stuff we dont need and dont want.
Nobody is preventing anyone from using the pool, People are choosing not to use it.