I think envy implies, by definition, that your desire is, to a certain extent, at the expense of your happiness for the other. If I am fully content with your having it, and simply want something I don’t have, which I happen to know about by seeing it by you, I am not envious, simply desirous. If your having it fuels my desire, that means I am not happy with your having while I don’t.
The concept of Ayin Horah (which def. exists, no matter how you deal with it), is based on the acknowledged difficulty we have in truly “fargining” others.