I’m fairly certain the Yerushalmi in Horiyos makes the exact opposite point that the Sifri makes (as brought down by Rashi). The Yerushalmi there states that you follow the chachamim *only* when they tell you that left is left and that right is right.
The Wolf