IMHO, people who don’t achieve a modicum of fluency (probably someone who came as an adult will never lose his accent) simply have a fear just as some people have fears regarding Math or other subjects. I had great difficulty with French in school but Hebrew cam easily even though the US State Dept considers the former much easier to learn than the latter. The two keys were motivation and immersion (and BTW, the fact that olim speak in English, French or Russian among themselves does not necessarily indicate lack of fluency in Hebrew – it just feels more natural). I also find Hebrew to be a remarkably logical, regular language unlike French, which has many exceptions.
People who come as retirees can get along without it but others will never feel part of Israel. It will, of course, also severely limit job opportunities as with anywhere else.
As for Yiddish, unless one is a native speaker it will be very difficult to learn. It is more or less a form of German and his experiences studying the latter drove Mark Twain to write an essay called “The Awful German Language”, which can be read on-line.