If you were so involved with what he was doing…how come you weren’t learning instead of watching him. Sounds like bitul Torah.
And certainly you should be dan him l’kaf zechus.
In fact since he was looking in the sefer and you were watching him, he was probably learning more than you were.