If having to face that person would be a cause for stress for you then its probably not worth it. If you can put it out of your head and not think about it then go back. (A lot depends on what was said/who said it/how it was said.)
Was this a group for purpose or for socializing? (You used the word group which sounds like a group getting together to learn something etc -and shmoozing does have purpose)
RABBAIM brought up a very good point, btw.