The sentiments expressed by the OP are both legitimate and natural for a Torah Yid.
Something not being assur, does not mean it shoudn’t be avoided. The question is when to avoid what is technically legal by the book, yet against the spirit of the law or tradition or common sense.