For something to be assur behanaah it has to be Derech Bishul, I don’t think a microwave counts.
If the microwave was dirty and the Basar Bechalav dirt came in hot contact with the food being warmed up, it could possibly be Assur Behanaah.
If the kosher food had 60 times the treif, it wouldn’t be Assur Behanaah.
In general, something must be an Issur Deoraysah to become Assur Behanaah, which isn’t always so easy.
If the previous Basar Bechalav was an “Issur Baluah” (for example a potato which absorbed Basar Bechalav), it wouldn’t asser behanaah.
If it wasn’t kosher meat and kosher milk, (non kosher species is definitely not assur behanaah, kosher species is assur behanaah according to most Poskim, Dagul Mervava is meikil bmokom hefsed).
In short, it probably wasn’t assur behanaah.