It is assur to speak whilst wearing Teffilin AND between putting on the shel yad and shel Rosh.
I understand the question to be: May one speak once the shel rosh have been removed but the shel yad is still on?
Not that it is right, I have seen som people who raise their shel rosh slightly off their head so they can speak, and put back in situ when they have finished talking!
The story is told about the Rebbe in the city of Sanz who saw some bochurim talking during davening on chol hamoed. The story goes that the rebbe said it is better to wear teffilin on chol hamoed and not talk during davening, than to not wear teffilin but to talk. It appears that some people who are descendants from the kehilla of sanz have the minhag therefore that bochurim wear teffilin on chol hamoed but married men do not!
Can anyone confirm this story?