Gateshead, that’s crazy — I don’t believe people said such things. Like you, I’ve always loved maps, so I don’t understand it.
But I know that many people here in America have no clue where anything is. Even my wife (despite having traveled to various countries, including Belarus) often is unsure about where particular US states are on the map, at least until recently, when we started going over maps with our kids.
I would guess that 90% of Americans have no idea that Belarus is a country. Jews are more likely to know it’s a country, since many have ancestors from there.