Personally I'm going to be the discordant voice, rather than the father being a scumbag which is the typical situation I would prefer a normal guy and you could go towards the threesome type of relationship that ends up being overboard, the wife isn't satisfied on sex and trying to solve their sex problems they end up adding you into the equation with rules that are being bent bit by bit.
It's a fairly normal situation, it wasn't working out, they tried something different, it ended up as the final hit to their relationship, of course another possibility is trying to help the poor guy to actually solve his sex problems with his wife. I have to say that it gets a bit boring the typical stereotype of "all males except the MC are shit".
If you truly want an antagonist I would rather prefer that one of the girls have a boyfriend and that one is a dick with harem dreams that you have to kick his ass, but well, gonna leave it at that and good luck with your game.
RedPillBlues Comment or don't comment, but there is nothing more useless than reacting with a facepalm because someone gave his opinion in a thread that asked for it.