So far the posts have taken one of two positions: Either “Its my seat, he can get out”, or “Its bad middos to kick him out”.
I think that middos aren’t only on the seat owner. The person sitting there is also expected to have some middos.
Finding someone else sitting in their seat is upsetting for a lot of people. It leads to aggravation, and feeling bad if they do ask you to move. Just read many of the above comments if you doubt this.
I often daven in shuls where I don’t have a seat, and I always ask the people nearby if the seat is available, or wait until late enough in davening to assume no-one’s coming.
Having said that, using this as a reason to kick the fellow out because he has bad middos by taking my seat – that itself is bad middos! Two wrongs don’t make a right etc.