Yeah. I agree on this, it can be pertty annoying dealing with all that min-maxing, your focus shifts from the story too much on the points themself. And action of juggling one thing over the other in these scenarios are too much of a bother and leaves you quite frustrated sometimes.
Tho, "point system" is pretty great system of rewarding player for his attentiveness, engagement and social or just plain awareness of the situation. Like, what player wants to achieve, how well he followed the story so far, his judgment skills, how good he knows the characters and etc, etc, etc. But dropping on player the situations where he must choose one over the other out of blue, almost always break immersion into the story.
For now, VN plays like a "harem story"(tho, I'm not sure if it is one. no tags), it's not a bad thing or a good one. It's just sets expectations. For the most part, player goes into these games with goal of getting them all™ or for someone speecific in mind and 'point juggling' can ruin things for both.
At least, that is the consensus I'v seen on this site so far about these things. Maybe, I'm wrong.