“That’s what bothers me. These LOR’s are supposed to be wise. Why can’t they collaborate and come up with a common standard? Wouldn’t that promote unity among Jews? Is there a reason why there are so many groups of observant Jews who follow different practices?”
Yours is a very good question, and deserves appreciation; Why machlokes in halachah seems to be built into our system of mesorah.
1) HaShem not only gave us His Torah and mitzvos to reveal His ratzon in this world and to bring us to the next world perfected and ready/worthy of enjoying His Shechinah, but He also gave us Power of Attorney, so to speak, in this revelation process. This is referred to as Torah of the m’kabeil. Torah reveals His kovod. The more Torah, the more kovod. Machlokes between Gedolei Torah – Tana’im, Amora’im, Geonim, Rishonim, Achronim, Poskei Ha’dor – increase Torah, and therefore His Kovod.
2) The Torah is for all Jews. Not all Jews’ needs, level of service, availability of needed items, social environment, etc. are the same. The Torah still needs to be for them. It is for this reason that machlokes between poskim of different communities and/or countries differ from each other. Some countries did not change into Shabbos clothing for Shabbos Chazon (during the 9 days), while others did/do. It has been explained that those who did not, lived in countries with vibrant economies; it was Shabbosdik all week. They did not change in order to feel the 9 days. Those who did change for Shabbos Chazon, livedbin poor countries. They changed in order to feel Shabbos (it was 9 days all year for them). Is this a a machlokes?
Rav Gustman ztzvk”l was fond of saying he first evaluates the sho’el (the one asking the Rabbi a question), then he evaluates the question. Does this mean there is discord, disagreement, many Torahs? Absolutely not. It means the Torah has a perfect road for everyone to get to their Olam Ha’ba.
3) Related: A great loss of money is sometimes a valid reason for a lenient p’sak. A wealthy person may not “deserve” such a ruling. Is the Rav who gave the lenient p’sak to his poor congregant out of sync with the Rav who ruled strictly for his wealthy mispallel? Should the two Rabbis have gotten together to reach a compromise so neither member should have received the proper ruling?
This is why it is so important that everyone have a close relationship with a Rav who knows him/her, their background, goals, family, health and financial situation, etc. This way everyone has a good chance of keeping the entire Torah properly and earning their deserved place in Olam Ha’ba.
I would add that there is much more to be said on this important topic, and that perhaps even what I wrote above is not the perfect answer for everyone.