This is a custom of good manners that seems prevalent in almost every tradition and society on earth. I have no idea why someone would say that you shouldn’t say it. It’s even in the Gemara.
The Halachah is that you can’t say it in a Beis Medrash though. Maybe that’s what they were thinking of?