“Tur Shimon used to provide three hundred loaves of bread (for the poor) every eve of shabbat. Why then was it destroyed? One says, due to licentiousness. Another says, because they used to play ball.”
The Yerushalmi provides us with no further clarification on this matter, but it is clear to me that chazal are associating destruction with chillul shabbat, and that ball playing is seem as being out of line with the proper atmosphere we should be aiming to inculcate on shabbat.