Interesting to note that we davened to day for the Kohanim to return to the Duchan, for the Leviim to return to their singing – no mention was made of anyone else performing any Avoidah.
The Or Hachaim brings it down in a couple of places (Vayechi 49.28, Bamidbar 3.45, :
http://hebrewbooks.org/pdfpager.aspx?req=39995&st=&pgnum=20
http://beta.hebrewbooks.org/pdfpager.aspx?req=38295&st=&pgnum=143&hilite=
I can’t remember where the Daas Zekeinim is.
See also Moadim V’Zmanim Ch”7, Si’ 169. (Not on Hebrewbooks.org)