I do think this is interesting. Of course, I'll need to see it play out, but one thing I've always said is this: There are only two rules for a Harem.
One: The girls are okay with the MC having the Harem. (And even this can take time to get to, a little inter-harem drama is spicy sometimes.)
Two: They don't sleep around with others. (inter-harem stuff *can* happen with MC approval and knowledge, it's just not usually my cup of tea.)
Anything else is up for debate. They don't all have to live with the MC or anything else. Just those two things.
I think that
Sorcerer does a really good job of this idea. All of the women have lives and personalities of their own. They are all powerful, independent, women. Some even live in different countries and entirely separate realms of reality. But they have all committed to their relationship with the MC. Now even in that game, not ever girl is in the harem by the end, so I'm not saying that every girl *has* to be in the harem. A few girls in that game are set up to join in a possible sequel and a few just didn't work as full harem members (notably a pair of gilfs that game has just wouldn't fit with the harem at all). Still, it always makes me a little sad when a girl doesn't get the full package.
Okay, so it's a true spinoff that one of the My Dorm characters make a cameo in, but isn't a focus in. Got ya. I'm not usually a fan of Female MC games, they usually turn into cardboard cutout "Turn girl into whore" games. But you've done a great job so far on My Dorm making the characters seem realistic and believable, so I'd give the game a try.
So, if someone isn't on her route, where does the idea for the harem blossom, I wonder? Which girl besides her would be the first to really pitch the idea? You don't have to answer, I'm just thinking out loud.