I think there are quite a few people here, and not only here, who consider harem only when all the characters, at least those considered LI and fuck-buddy, are included in it. If there are fewer, it's considered a triple +++... most of the time. As far as I'm concerned, I've visited a few, and in my experience, it's not the number of women who make it up that defines it. The number can vary greatly depending on one or the other, as do the practices depending on the region of the world where it is located. But in all the cases that I have observed, the harem is, to begin with, a place reserved for women, concubines and others. The connotation of slavery is often strongly imbued there. The women occupying such a place, although having different tasks assigned, are nevertheless servile towards their "master". So the master says, the woman does, for her happiness or her misfortune.
So it has nothing to do with what we players see in these fantasy games