If I can assume that “bummy” (which is not an English word) is supposed to mean “exhibiting characteristics of bums,” I would have to assume that the reason smoking is considered bummy – an assumption that you have made – is because there is a higher percentage of bums that smoke than the percentage of smokers amongst the population at large.
Perhaps when people refer to smokers as “bummy” they are speaking of the result of the smoking – the stench that rests on the smoker and his surroundings – and not the act of smoking.
EDITED POST
Please keep your language PG appropriate.