It has been said that the reports of such goings on may have been embellished by British explorers in their writings.
They definitely were. Europeans liked to talk up things from "The East" and make more exotic stories out of stuff from Asia and the Middle East.
Harems really weren't debauched sex parties. They were mostly dull luxury prisons where the wives and concubines of the ruler (many of whom were frumpy middle aged women if they'd been there for a while) lived with their children to keep them safe from being corrupted or killed. And lesbian sex - while possible - wasn't very likely because political scheming was more or less constant. Nearly every woman in there with you was your enemy. "Accidents" weren't even necessarily uncommon as one woman might try to minimize the influence of (or the offspring of) her rivals. The Ottomans were notorious for that sort of thing - it wasn't uncommon for a newly crowned Sultan to have most of his siblings murdered.
(Which some people point to as one of the reasons behind the gradual weakening and eventual fall of the empire. Children who have literally never seen the outside world who had lived in complete and utter paranoia for most of their lives - and who may have had to murder their brothers - tend not to be the most stable people when you plop them on a throne and tell them they're in charge now.)
But yeah, if you were living with people who would happily murder you and all your children the moment the ruler dies and they manage to get their own son on the throne (if not
before), it's not really all that conducive to sexy times.
But that's depressing. So European writers preferred their version where all of the women in the harem were nubile young beauties who were constantly sexed up and willing to fall into the arms of any man who could manage to sneak his way in (and who would just make out with each other when no men were around).
Come to think of it, I prefer their version too.