I once asked my father-in-law, who is a Rav, the following:
We say b’rshus Kohanim, because really, the kohen should lead the zimun. If someone else is leading it instead, he asks for permission. So if a kohen is eating at someone’s house, and is asked to lead the zimun, he’ll often say b’rshus baal habayis. But if the baal habayis leads, he’ll say b’rshus hakohen. So who actually has the right to lead the zimun? Is it the kohen, or the baal habayis?
As an aside, when I was in yeshiva, there were some guys, who if they had eaten out, would say b’rshus baal habayis v’baalas habayis. This is wrong, because the woman is unable to lead the zimun, so you don’t need her permission.