Not really. You are not allowed to hold your moneybelt or shild while davening, but you are allowed to hold a moneybelt or child any other time.
You are not allowed to daven while in a bathroom, but you are allowed to be in a bathroom while not davening.
So, Maybe, you are not allowed to look at a little girl while davening, but are allowed to look or touch a girl(unless its creepy, cuz then you might scar her and you might end up in jail)while not davening
Sam2: Reading your two comments here, it seems you are under the impression that men may touch girls until they reach puberty even though they may not even look at the upper arms or legs of the same girl over age 3 (per the Chofetz Chaim) or 6 (per the Chazon Ish).
That would be counterintuitive, wouldn't it.