The source for a limitation begins with the Gemara in Avoda Zara (8b) which comments on the Mishna’s statements that during the three days surrounding a gentile holiday one is not permitted to engage in business with a gentile. The Gemara then notes the pagan holidays and includes Starana and Kolandarum which, the Gemara associates with the eight days after, and before Tekufas Teves. The Rambam (Pirush Hamishna) associates these days with the early Christians who, it seems, might have stolen the concept of these holidays and their timing and created a holiday season surrounding Christianity so that the beginning of the secular new year would be associated with Yuska and his birth, adding importance to his existence (See Beis Yosef to Orach Chaim 117 in the name of Rav Dovid Avudraham). According to this position, the entire holiday season, from Christmas eve to New Years, should be a time not to engage in business with Christians.
Yet, the Rashbam (cited in Talmidei Rabbeinu Yonah A.Z. 2a) notes that the custom was only to limit business on Christmas and Christmas Eve as these were the only days that had any significance on the calendar. Yet, this speaks to engaging in business.
A source for a prohibited night of Torah study appears in the name of Rav Zalman Zvi Ophauzen (Otzar HaVikuchim, p. 174) who was chastised by a Christian galach during a debate about the “well known” Jewish practice of not studying Torah on Chistmas eve. The Chavos Yair (Mekor Chaim, Orach Chaim, 155) notes a practice not to study Torah on the night of “Holada”.
In Minhagei Yeshurun (cited in Taamei Haminhagim p. 500) it is suggested that the Jewish Battei Medrash were out in the fields. If Jews were to be going to learn on those nights, they would be subjugated to potential attack by the gentiles coming home from midnight mass. To prevent the chaos, Jews were told to stay home. Otzar Minhagei Chassidim (119) puts the same idea in a different light, suggesting that it was the Christians who shut the lights in the homes and Shuls of the Jews not allowing them to study.
The Chasam Sofer (Shut Chasam Sofer, VII:31) quotes his Rebbe, Rav Nosson Adler who felt that it was forbidden to rejoice during the holiday. Therefore, Torah study, which gladdens the heart, cannot be studied. Rav Moshe Feinstein (Iggros Moshe Y.D. III:85) applies the same logic in explaining why schools must remain open during this season. The Chasam Sofer himself argues with his Rebbe and suggests that the Minhag of not learning on this night was to avoid giving power to the Sotan. This idea is advanced in the works of Chassidic Rebbes (Bnei Yisosschar’s Regel Yeshara Erech Yeshu HaNotzree) where the idea of Torah study on this night is associated with Sakanna.
The word Nittel itself probably come from an acronym for Nolad Yeshu Tes Lteives “Yuska was born on the ninth of Teives.” Another possibility the word Nittel etymologically comes from the Loshon Kodesh natal (to have been hanged.) Or it can also be come from the Loshon Kodesh word for “being taken away”.
BTW the reason for the mourning on this day is due to all the Yidden killed on this day in Jewish history.
Someone was once asked to eulogize Theodor Herzl. After a few moments, he came up with three positive traits: Herzl had never spoken while putting on his tefilin, had never thought about Torah matters in unclean places and had never studied Torah on Nitel.