However, since most Rishonim only bring the sevara of chasnus, and this does not apply to a mumar the gezeira would not apply.
This is an oversimplification. There are sevaros, said by big acharonim (as mentioned earlier in this thread), to be machmir even according to the rishonim who hold the reason for the gezeiro is chasnus.