Sam, your post is misleading.
There is solid basis to say that negia is not d’oraisa when it is not “derech tayva v’chibas bi’ah.” This, by the way, is clearly not the same as simple affection.
See the Shach 195:20 found here who says this straight out: http://hebrewbooks.org/pdfpager.aspx?req=9146&st=&pgnum=316&hilite=
Many contemporary poskim happen to agree with the Shach, including the noted machmir, the Badei Hashulchan.