Gavra, actually the perush is Rashbam not Rashi. he is correct that the diagonal of a five by five square is similar to that of a four by six rectangle (the former is the square root of 50 and the latter the square root of 52)). I do not understand how Tosafot made the mistake that the former is greater then the latter, especially as they know that the the area is greater. They are also very close regarding the diagonal of the four by six (the difference can be put down to a scribal error). Chazal definitely knew the Pythagorean theorem as in several places they say that the diagonal of a one by one square is 1.4.
In any case, this is not Halacha but Geometry.