While it may not be realistic, citing other games is not proof that it is wrong I'd argue
But on a more technical level, each point of fitness takes away 1 hour waiting time, so with 50 you have taken two days off. (Also each point of sexperience shortens the time by 30 minutes). With the base being 5 days (120 hours) that's not enough yet. However, going to 100 fitness isn't a big deal, and 20 hours usually is enough to go every day, plus you'll get some sexperience along the way. (Also remember that there is fitness enhancing clothing)
If you come to NG+, both fitness and sexperience stay, so you can go all in like the horniest of teenagers from the beginning.