You are correct though that there are cases in which it would be hard to tell which is the chumra and which is the kula. There is a funny Gadol story about that, but I forget how it goes. Maybe someone else here knows it and can tell it. There may even be more than one.
Usually, if someone is NOT ALLOWED to do something vs. being allowed to do it, or if someone HAS to do it vs. being allowed to do it, you would call it a chumra.
The confusion sets in when one group HAS TO and the other group is NOT ALLOWED to. Then there may be differences of opinon regarding which is the chumra and which is the chumra. Likewise, if one group has to do one thing, and the other group has to do the other.