Fkelly,
You are right, but the reason is because in fraternizing with them over fine wine you might learn from their bad ways/marry their daughters. If this is the reason, then it should not apply to someone who is frum in public because other people will not know that they are not frum and there will be nothing to learn from.