Oh wow I always do that after the guy who has the best chair leaves. But I always make sure to return it before i leave so he doesn’t know. No harm done.
In your situation: Is he a veteran in your office? Are you the rookie at your job? Who’s older? I’m pretty sure these things matter.