I feel that they're should be some sort of cool-down period after missing a date. Have the girl be mad at you for a couple of days, not talk to you, take a harsh stat drop, and recover a small piece of it two or three days later, apologizing for missing the date. Then things would return to normal.
Also, is there any stat hits for actually BEING sick? It should mean something more than just dialogue. Not being so good at work, or maybe activities taking more out of you than they should.
Just spit-balling, but would help a little bit, especially with the dates thing and if story-driven events force you to miss dates.