I get what you say, but I am not sure that I would call this having us by the balls, he is paying us more than Eva and his other workers, even if we work more than anyone that's a win in my book as our main fear Ch1 was not having money.
I would understand his point if he secretly liked his wife going with others and he wants to help, but if he is actually angry at us, it was as simple as telling every employee in that small town that we are not reliable so we can't work everywhere, he even said that when he threatens us, the other option was letting us work but paying really low wages.
But yeah I will probably have to turn my brain off if I continue playing cause there are other instances where I was thinking I would never do what he is doing either