It's a terrible design choice. Consider the big majority of the playerbase for these games are males, and they are aware of it or not, having a charact you can identify as makes you want to play further, specially since most of these games are visual novels with very little gameplay elements.
Couple that with the previous game made by the same dev, which is a male MC, and that's why people is loud. Not the first time it will happen and not the last time, happened to Glamour after Dating my Daughter and look how that is going, specially compared to DMD. It's not even a matter of taste, but content. Sure, you are not going to make the same game over and over ICSTOR but there are things people expect from you when you have to drop a game and make another to save it, and instead of making something similar, you basically do a whole 180.
Don't get me wrong, mate, I'm well aware of this, and saying the manor is dead is too harsh, I've been following the news after the whole thing went down. But this is simply not what I and I bet a lot others were expecting, and I doubt the people that supported you for the manor would support you for, well... This. And you are not the first one to do this. As previously mentioned, it happened to Mr. Dots, and even though he is doing relatively fine still, it is nowhere as close to DMD, same with smaller developers like Sex Curse.
I wish you luck with both your games, mate, and it's a shame what happened to you, but even though my opinion is irrelevant, as shown before, the amount of people following you will drop, not because this new game is bad, I don't know, I won't play it, but because it's way too different from the previous in premise, so you basically have to make a brand new following instead of retaining the previous one. And I don't mean to be mean or anything, just stating this, since considering some other reactions, I bet I'm not even the first one to make this remark. It's up to you to consider it if it's a "loud minority" or something else, and what you do with it.
Again, sorry if it sounds rude, and again good luck.