I always considered it male tznius- colored shirts grab attention; when a person is dressed in black and white it seems less noticeable and allows you to focus on things that are more important than fashion. Black and white also looks a little more formal, a reminder to not slouch or appear lazy.
Like others have said, it also allows you to identify as part of the community, separate from the rest of the world.