I'm not saying it makes it any better for the player necessarily, but to be fair, the wife has no idea she's married to the MC. So she's not with the other guy in an effort to cuck him or get revenge or anything. I know a lot of players still really hated that, but it's technically understandable in the situation of everything when the Paradise leaves. But for some, they still hate that the wife is with another man, and for some the non-MC sex still happening is too much even if not seen.
New players should be aware of that before playing, because that may or may not be something they're OK with in the game. Also, that is mainly in the first bit of the game. Once a certain event happens, you don't really have that even mentioned again until the very last part.