Thanks for clarifying it, dev, but I don't see the appeal in a so called "semi-harem" in this case. Either go full or let it be. Harem fans tend to, at least in most cases, to want them all or they won't bother with the game. And most players won't play a VN more than one time either. So the argument for it to have more replayability isn't working, because it changes only a small percentage of the overall experience, which won't be enough for them. Mind you, I said most players, because there are always exceptions to this.
And just as a disclaimer, I just wanted to share experiences I made over the last few years interacting with other VN players. This isn't meant as a post to change your mind over your plans. You do you, you're your first player and customer and if you won't like your own product anymore, because of suggested changes from the community, you won't put your heart into the development anymore and the player will stop liking your game as well. There will always be someone, who doesn't like your VN, but don't let yourself discourage from it.