the chareidim dont stop the dati guys from sitting and learning all day. by all means let them do so. were all for it.
and you must show respect for all talmidei chachomim. even if he isnt your rov. it is bizui talmidei chachomim to publicly pronounce your lack of respect for them.