Swinging is, by definition, not netorare. it is not intended to cause jealousy.
And if we used your definition of NTR being based on what the people playing it feel, then any game with a second male in the game would be required to have the NTR tag, because someone, on these forums, will feel they are being NTR'd.
Unfortunately most harem players feel this way which causes a lot of negativity on these forums.