About 'harem' I'm not sure about that either anymore. I know it is in my planned tags, but like you said it would be troublesome to make it work. However, I haven't discarded that idea completely yet.
As much as I like to stay close to reality, this is still a "fantasy" story and maybe not everything might be 100% realistic, but I'll try anyway.
Anyway, I thought of [my points against harems] and I do have in mind what you just said, but I can't, or more like don't want to, elaborate further onto that without spoiling