Are you sure? I'm on day 37 (just looked it up) and I reloaded when she asked me whether to meet her at night (where she asks you to become her boyfriend). As such, when she asks me whether I have time, I can just say no and nothing happens aka I'm deferring my answer.
That's unfortunate to hear. Getting those charisma points is easy, two I can already get and Evelyn and Callie are basically done. Samarra, I guess, has something to do with the other points I'm still missing (2 agi) but IIRC, I need to use some spell during the fight to land a hit...or at least that's what was talked about. (No spoilers, please.)
So either you screw them over by pretending to be their boyfriend and then denying them at the end and they are sad, or you deny them immediately and they are sad (Primrose certainly didn't take it well _at all_ when I declined her offer, before reloading and ignoring the whole thing). So you have to be mean to them either way, that's....not ideal.