As I understand, elevators are designed with a counterweight equal to the elevator filled to ~40% capacity. The elevator is not actually “lifting” or “lowering”, rather just sliding one side over to the other in a mostly balanced system. There may still be some additional strain on the motor, but nothing close to the force required to actually lift a human being.
In an empty elevator with a single rider, the motor would have to hold back the force of the weight, which is attempting to pull the elevator up, whereas lowering the elevator in the same situation would require force to lift the counterweight, but not the person. If the elevator was filled above 40%, the exact opposite would be true. Raising the elevator would require more force, and lowering would require less force.