Take a look at the Gemarah at the start of the fourth perek of Menachos; as you study the sugya with the rishonim (or at least when I did), and carry it over to the primary halacha codes (Rambam, Tur, Shulchan Aruch), the follwoing framework seems to develop:
1. M’dioraisah, there is a positive obligation to wear tzitzis IF and ONLY IF you are wearing a qualifying four cornered beged during the day time.
2. If you wear a four cornered beged without tzitzis during the day, most opinions hold that you have been m’vatel an asei (failed to perform an obligation) b’shav v’al taaseh (passively rather than actively), but NOT that you have been over an issur aseh (an prohibition that grows logically from a Torah obligation, but which is not spelled out in the Torah), and certainly have not transgressed a lo sa’asei.
3. According to only one rishon (I believe it’s the Ran, but will have to check), wearing a four cornered beged without tzitzis during the day is an issur asei.
4. No rishon holds that there is an obligation to wear a four cornered beged in order to wear tzitzis, or that the failurre to wear a beged that obligated you in tzitzis is any kind of problem – not even a passive bittul aseh.
5. That said, it has undoubtedly become a minhag (perhaps on the level of minhag yisrael) to wear a four cornered beged in order to be obligated to and to actually wear tzitzis.
6. The minhag (yisrael) to wear tzitzis should not be mistakenly thought of as normative halacha – even d’rabanan. It is a minhag, and like most minhagim (depending on there strength) it may be set aside for individual special circumstances. Thus, the common practice of not wearing tzitzis while play sports, gardening, or doing pother strenuous work that causes one to sweat. I have also heard of p’sakim not to wear tzitzis given to people with even minor sensory issues for whom wearing tzitzis is unusually uncomfortable such that is disturbs their ability to go about their daily routine.
7. It is therefore difficult to understand the views of some robbonim/roshei yeshiva who expect tzitzis to be worn at all times regardless of what a person is doing. However, such views may be explained by those rabbnim seeing a particular need for tzitzis to be worn as a means of social cohesion and constant connection to Torah, regardless of whatever else a person may be doing (work, sports, ect.) – which after all is exactly what the TOrah tells us tzitzis are for.