So let’s go a step further. Maybe it’s saying that it’s about the attitude the whole time.
The Ro”sh (Brachos 11b) says that it is sufficient to say Birchas HaTorah once a day even for someone who works, because he rushes to finish his work, and is mentally involved in his learning the whole time.
Surely such a person, although a “balabos”, would qualify as a “yoshev bais hamidrash”.