I don't really believe Jill went too far off track. She has one main aspect where she's out of her comfort zone, and that's dating and sex.
She doesn't have any issues having male friends (she treats Tybalt as one), but when things turn to romance, she turns to butter. That's been consistent from the very beginning.
The Tybalt blackmail isn't a fantastic plot element; we have to accept that she actually believed Tybalt when he said Trent wanted to press charges and Tybalt was trying to dissuade Trent.
But Jill has known for ages that Tybalt is interested in her (in episode 3 she tells Bella she knows Tybalt is trying to win her over).
So when Tybalt asks for the three dates, she kinda relents, like he's doing her a favour by stopping Trent from pressing charges, and he's asking that Jill go on a few dates with him. Sure, we hate Tybalt, we've seen him at his worst. Jill hasn't, so she gives him the benefit of the doubt. She only later realises the error of her ways.
But getting back to Jill and sex, in every scene when the mc makes a move on her, she typically has trouble. Even when Bella mentions dating him (in the same scene that she mentions she knows Tybalt is interested), she kinda baulks:
Jill: "Date him?"
Jill: "I'm not sure I'm looking for someone to date..."
Jill: "Did you think I was?"
Bella: "Well, you're a woman. Stranger things have happened."
Jill: "Haha, yeah, but still..."
So yeah, i think the blackmail plot is viewed differently by us (who hate Tybalt and completely understand his motives) and Jill, who treats him like a friend who she knows is interested and wants to be kind to his feelings. When she blows Rusty off in that flashback, he's basically a stranger sleazing on to her and she shoots him down. Tybalt isn't the same thing, even though
we hate him more.