No, Kollel is NOT a luxury, it is a necessity.
Joseph, LEARNING TORAH is not a luxury. Kollel is a luxury. Please understand the distinction.
To build a bit on what Oomis said – I have a few friends that LOVE learning. They learn everything but really love delving into gemara. They are looking for a true lamdan who can learn with them. For them, learning is NOT tiflus.
I would compare it (on a very imperfect scale) to Calculus – I loved learning Calculus and many of my friends wouldnt touch it with a ten foot pole (many had to take it as a requirement and hated it).