You only feel the way you allow yourself to feel. Don’t let the small-minded people drag you down. The only judge you are beholden to is Hashem, yourself and, to a lesser degree, your spouse. Other people (your rov, children, friends) can offfer their input, suggestions and guidance to the extent that is appropriate to their role, but in the end its your nestegg your building so whatever you expect to have sacked away is up to you.
That said, outsiders (as meddling and critical as they are) do provide a valuable service. They prod you to go beyond what you might think you are capable of and they give you a chance to self-evaluate yourself to be sure you’re still on track to achieve your goals.
Don’t let them bring you down, let them build you up!