The origins of the word 'Harem' are from the Middle East and Asia and refer to cultures where the wealthy and nobility not only
could have multiple wives and concubines, but were expected to do so. Male servants and guards for the women of a harem were often eunuchs (castrato in the Latin/Roman scenarios that emulated the Eastern harems). After all, not every fat, overfed and over-privileged show-off with 40 concubines could possibly give them all satisfaction, and they hated the idea of NTR as much as some of the vocal folks in these forums.
Technically, members of a Harem are not just people you have slept with, or could continue to have sex with, but people you 'own' and have exclusive rights to (even if you might then choose to lend someone one of your concubines for a night).
In games, the closest true definition of a harem is where you end up with multiple girls who are not just occasional fuck-buddies or friends with benefits.