“But 95% of the people complaining about us aren’t Chassidishers. There are tons of people, who are more Maikil than Lubavitch in pretty much everything except for Zman Tefilla and possible Tznius”
This is the problem, you think Chabad’s chumras in other areas justify laxity in the halachas in these areas. Meaning like this: “Chabad has beards, keeps pas yisroel, wraps Rabbeinu Tam tefillin, etc. Therefore, if someone doesn’t do these things, they can’t criticize us for our kulas.”
These chumras are middos Chassidus and we respect them, but don’t hold them to be meakev. Zman hatefillah, fasting before shachris, sleeping in the sukkah, who counts in a minyan, etc. these are all meakev (at least according to all poskim prior to Chabad). So, it’s better for someone to always keep the ikkar hadin than to keep a mixture of chumras and assur hanhagos.