PBA- You have a good question but a distincition can be made as Chein said. When someone sees you wearing a suit they most likely can not know which brand you are wearing and don’t assume its a suit made with Shatnez; hence no maaris ayin.
Regarding YD 87, this wouldn’t apply to having a coffee with pareve milk after a flieshig meal because its so prevelant and one immediately assumes its pareve milk. However, to drink pareve milk straight up during the meal is a problem because that is not common. See the link I posted above- Thanks Mod!