Shtickey, the Ramban there points to the Pasuk that says explicitly that the count is besides for their wives.
Rashi asks about the Te’umos and answers that the Te’umos must have died. Rashi obviously learned in the words, ???? ??? ??? ????, that we are not counting the wives they got from the local population. Therefore, the Te’umos should be counted. Yosef, however, did not get sold along with a Te’uma, and perhaps Rashi is going with the Medrash that Asnat was Potifars daughter, or as you said, she died, too.