Korbanos are a whole Parsha for themselves. Also, although he explains that Korbanos is a Kosher alternative for Avoda Zara, it is still not a branch of Avoda Zara. Switching clothing, he holds, is part of the Harchakos of Avoda Zara.
More important, although the Rambam generally refrains from mentioning his reasons from Mora Nevuchim in the Yad, what we do see from there is that he did not see this Issur as a Harchaka to Arayos like Rashi did. To him it is a Chok. In that sense it fits right in with Klai’im.
I agree with The Wolf. It should not be called a stupid, nor theoretical, question.