I have a feeling that this might be the root of his anger. Obviously, he didn’t randomly throw you out and bash you. He thought you did something. You don’t think you did anything wrong, so even your apologies were undoubtedly insincere; how could they not be, if you don’t think you did anything wrong! So you’re in another tough situation, even worse than the $50.
Good luck getting through them.