No, it's the same if the woman she sleeps with is not a part of your harem, and if she is:
Multiple people engaging in sexual relationship with each other is the definition of poly, if girls in your harem sleep with each other it is poly.
NO OFFENCE but.
The very definition of a Harem is mutiple Women dedicated to only one man to the exclusion of any other males!.
But at no time have I seen it either said or writen to the exclusion of any other woman either inside or outside the Harem.
So unless you have seen diffrently then again NO OFFENCE but working on that baseis alone would mean.
You can LET your girls sleep with any other girl inside or outside and it will still be classed as having a Harem.
Polyamorous however is a relationship in which a woman or a man can sleep with any other member of that relationship be they either female or male or even mutiple males.