After playing it through, I was left the opposite impression. Almost everyone he meets wants to jump his bones but are too afraid of the consequences (his being the only dude alive sort of puts him on a holy pedestal). Others don't really care about it and just want to take him for a spin but are too shy/awkward to ask, and he's too much of a spineless wimp to make the first move, as male harem protags usually tend to be, so he's left in this sexless limbo until magic comes into play. Magic has sort of become a simple plot device though, because he really didn't need it, especially not when even the freaking queen wants a piece of that male salami without magic being involved.
It becomes infuriating to watch sometimes.