The above-mentioned remah doesn’t have the strongest of sources. (It comes from a nemukei yosef which learns derech eretz from an agadeta Gemara.) However, the remah continues and says that this is the accepted halacha, and poretz geder yishchenu nachash.
It would seem that if there was a specific situation where specific communities would have a minhag otherwise (for example, holding hands after a chuppah), the remah wouldn’t be so strong against it. (But if someone isn’t part of that community, he wouldn’t be allowed.)
