You won't ever miss an event involving a single girl as those events unlock as you level them up. There are a few scenes (threesomes mainly) that require two girls to be a certain level to unlock, so again, you won't miss anything.
BUT! and it is a huge BUT: every Sunday, there is a special event that triggers regardless of your progression, involving all the girls of the Manor. If the girls are not high level enough, the even can go from tame to really sexy. I'd recommend leveling all the girls equally and not losing that much time. Have a look at the walkthrough if you do not want to miss anything.