@Avreimi this isn’t about being “scared”. You cannot put a stumbling block before the blind, it is assur for a man to get any enjoyment from a woman he is not married to. There’s many halachos and many rabbanim who have spoken about this, and in any situation I suggest you speak to your Rabbi and see what he says.
Most of the rabbanim I follow hold that you should not say anything unless it is for a purpose. Unnecessary chit chat breaks down boundaries. What is it’s purpose? Think about it. As a woman I should not and do not want to have any relationship with any other man that is not a relative. What is the purpose of talking if not to create a connection? For what do I need to have a connection with a random guy?