So I’m thinking that maybe the Yakov Avinu thing needs some support- (although the Rabbi is a big talmid chochom), because some may class it as agaddata- and hence consider it non literal. But it’s not actually a part of the Gemara- it’s meforshim on the Gemara.
And there’s a Yalkut Shimoni that says the same as well.
Additionally, here’s an interesting analysis by The Rebbe on Yakov Lo Mes.
https://www dot hebrewbooks.org/pdfpager.aspx?req=15959&st=&pgnum=236