“But how do I know if people are just doing this so that I should feel good about myself or if they’re doing it honestly cuz they like my voice?”
Start a part-time business giving singing lessons (for women only, of course). If you get rich from it, that will prove that you’re a talented singer.