The main reason, is that it is not acting like an adult to not be honest and forthright with another person. It is worse to deliberately act in such a way as to make that person think negatively about you. No one is putting a gun to your head to go out with the person. As I said previously, there will be many such moments with which you are faced in life, where you will want or need to say no about something important and it may even be a really uncomfortable situation. This is part of adulthood.
(It’s why I am so against the whole “let the shadchan do the asking out and breaking up for you,” deal) If you are old enough to date, you are old enough to UNdate, as well. And yes, if you act in less than a menschlech way, people WILL talk about you.