Or worse, that if his brother is a real brother from his mother that we should be Choshesh that he might do Yibum which would be an Erva. Why does Tosafos only bring up a Zekuka marrying out?
Tiferes Yaakov asks this kashya.
The Maharsha might be helpful.