Daas,
It is not that bein adam l’makom isn’t important, it is that forcing private bein adam l’makom on a large scale is not what the Torah authorizes and it doesn’t do anything to force someone to shake a Lulav if in their mind they are doing nothing.
I am not saying that if they see a siman yaffeh they should be exempt. Under my proposed system it would require more than just siman yaffeh to make it into the quota. I am just borrowing the 5 years as an amount of time that is sufficient to allow us to judge the long term prospects of someone’s Torah future.