Pashuteh Yid, I retract what I said in my last post about a “printer error” and I apologize for it. It is most definitely not a printer error, and I was wrong to brush off your question in that way. Over Shabbos, I reviewed the Gemara and only then did I remember how to resolve this question.
The Gemara gives an approximation for measuring the diagonal of a perfect square as being 1.4 times the side. This is an acceptable approximation for the true measurement, which is 2^.5, and in fact this approximation for root 2 is used often by college students (as is 1.7 for root 3).
The issue at hand though, is not a perfect square but a rectangle, and the calculation of the diagonal cannot be defined by a single simple multiplicative rule (rather, one would need the Pythagorian Theorem, which is above the understanding of many, esp. in the dark ages). The Gemara requires that 8 amos be checked, and this is rounding up from the true measurement of approximately 7.2. In this way, the Gemara uses the rule of 1.2 times the longest side (or as the Rashbam put it, an equivalent length to the diagonal of a 5×5) and then some. It is using the approximation l’chumra.
Now the problem seems to be with Tosaphos, who says at the very end that a 5×5 must have a diagonal at least as great as a 4×6, because the area is greater. We know that this is fallacious mathematically. However, this sugya has nothing at all to do with teaching mathematics, and everything to do with ensuring that the property in question does not contain any carcasses. Therefore, what we are really concerned with is not drawing a diagonal across the chamber – but with checking a specific portion of the chamber to ensure that it is free of bodies. That is the reason why Tosaphos says that the chamber with the greater area (5×5) should have the greater checking requirement. He is saying that the Halacha should be that the 4×6 chamber can be ruled Tahor with less checking than the 5×5, because there is less area to check. Not that the true diagonal is shorter.
Ritva says this openly – he says that the diagonal that the Gemara speaks of is not straight but rather zigs and zags so as to make a complete 8 amos. In this way he resolves the 20 amos that are required to be checked as 6+8+6.