To be fair there what you said is true, there are guys who want the 'harem' only for them and no girl can touch the other BUT I'm not one of them I actually like when they all seem together in a relationship instead of waiting in turn for the MC to touch them.
That being said this does not feel like a harem, it feels like 2 couples are swinging together (mc and squire/wife and maid) and the MC seems just to go along with whatever his wife says.
I'm sure this was said before but the way you said it, it's not like how it is in the games so I felt the need to say it again for @
BeastMaster59 sake