You can disable the auto-invite function if you want but you will still be able to invite someone if you want them. When in the sex scene with with Smart UI press the I key and it will bring up the invite menu of all of the non-virgin girls, then you just pick who you want. Just keep in mind that the RAM and processor requirements shoot up fast as you invite girls. I have an i7 7700K and 3 girls slows my computer to a crawl, a 4th makes the game nearly unplayable.
I wish the swap feature was a bit more user friendly, some nice friendly buttons shouldn't be impossible, but every three-some is a fight with the camera.