jphone:
Correct. As the Gemorah and poskim makes clear, appropriate usage is to use it for the husband at home. Not for the street. (Also see the Shevet HaLevi on that point.)
This problem was widespread enough to cause the destruction of the Beis Hamikdash.