There may not have been kohanim from Aharon in the time of the avos, but there was a concept of kohanim- Check out Parshas Lech Lecha. Shem (MalkiTzedek melech Shalem) was a Kohen to Kel Elyon. Avraham gave him maaser from the spoils of his war with the 4 kings. Rashi says that it was because Shem was a kohein.
Also by Yitzchak, Rashi states that when he measured his produce and found that it had produced 100x more than he expected (Meah Shearim), he was measuring it to give maaser. Does not say to whom, but we know Shem was still alive. When delineating the mitzvos, the Rambam attributes maaser specifically to Yitzchak. (I don’t know how he explains the above episode with Avraham).
Mechiya amalek- we can’t do this mitzva now either in its literal sense. We fulfill it the best we can, for example, by reading parshas zachor and “klapping” out Haman’s name during the megilla. The concept of amalek exists even when the nation itself does not. I would suggest that perhaps the Avos were able to fulfill the mitzva in a non-literal sense, by blotting out the concept of chance and denial in Hashem’s hashgacha pratis tha Amalek represents?