Largely they wouldn't... However there's always a vocal group of pointlessly jealous insecure idiots.
The fear, is fear of them having sex with a harem member. be it current (at time of meeting) or potential future addition.
Seriously, just go through the thread and look at all the times people were getting seriously bent out of shape over the succubi (Yara and Qum) at the first Gathering - fucking with orcs (including Orcent) and other succubi.
The problem is also that SL ultimately made a harem game where the vast majority of the early game and setup is predicated on the game saying "this is not a harem". Most of the women that join the "harem" early on do not do so under the pretense of being "fully commited and exclusive wife #12 for Simon", most of them are not really signing up to be exclusive to Simon in a "harem-like" relationship with one-sided exclusivity. Also at the same time, SL simply wrote Simon to be very respectful of the wishes and autonomy of his partners, so she wouldn't write Simon to ever demand or request exclusivity if it wasn't something he was willing or able to reciprocate.
The result ends up being that SL needs to write the story in a way that the women in the harem just never
want to sleep with any man other than Simon, which ends up with almost all male characters being uninteresting, evil, sexually unavailable, or deceased, so that there is never a reason for them to even think about it. While I definitely do not think it reflects the intentions and thoughts of the author, I do think that, combined with SL's feminist tones in her writing, it can sometimes accidentally result in some misandrist worldbuilding, or at least be perceived as such.
I think the game could have had more interesting male characters and also kept them sexually apart from the harem, but such a thing would probably also need an explanation that may also not have been what SL ideally wanted for the story. I think she just had to compromise somewhere and she chose to not have interesting male characters rather than anything else.