I've thought about this one before. Quite a few times, in fact.
For me, it figured like so:
It really depends on how the hentai work is portrayed.
For instance, I cannot stand most representations of actual raep of my favorite characters. It simply bother me.
And, on characters that I've grown absurdly attached to, occasionally a well written doujin will feel like voyeurism, and put me off.
On the other hand, even for characters that exude the "Do not touch; Fragile" feel, I've seen a number of doujins that are done in either a "Loving first time" or First Person Perspective, and those are usually just fine.
Hell, FPP doujins are best.
So,, I guess what it comes down to, in the end, is how willing you are to see something you have an emotional connection to be raeped horribly by someone/thing.
For me, not really. For others, fap away.