As someone who developed a nut allergy later in life, I think you need to give your doctor a LITTLE more credit then you’re giving.
And I also had the exact opposite of what happened to you-I went to one allergist, he wasn’t sure so he told me I needed to get a (non-insurance covered) blood test, which I ignored and then went to a different allergist, who took one look at the results of my test and said, you need an epipen with you. ALWAYS.
Allergies can suddenly develop later in life, and the way your body works is (sometimes) the allergy develops over time. Meaning, the first time, you get a little tickle in your throat, or a few hives on your hand, but the allergy can develop into something life threatening if you keep ingesting it.
Go for a second opinion if you doubt your doctor. But to just brush off what the doctor says is foolish. And so is not having an epipen.
My mother was at a dinner once and there were nuts in the salad and someone went into anaphalactic shock, and he didn’t have his epipen with him because he was too embarrassed.
I keep mine in my purse with me at all times. My bosses and co-workers know what I am allergic to, and have made themselves fully aware of where I keep my epipen so that they are ready to help if G-d forbid I need it.
It’s worth it to make sure that you aren’t allergic to foods. You never know in what foods things are hidden.