I remember asking just this question from some menfolk on Yom Tov… a lively discussion ensued… and if I am not mistaken it was brought up in some sefer… (I may sound like a total am haaretz here, please forgive me) about in the bais hamikdash (or was it just a huge shul?) where it was so huge, that the people didn’t really HEAR the bracho, but a flag was raised so that they can answer at the appropriate time… Hey, maybe someone can fill in the missing pieces here…
In any case, with the answer being no… I still feel uncomfortable NOT to say AMEN…