Because you are using logic and common sense and trying to get there by being open and honest with them both. That's a mistake here.

This is a rough guide of how to get to the poly ending.
1. After first sex with Mallory, tell her it's just to help her.
2. Do romance path with Jaye
3. When Jaye is upset choose "what's wrong" and then "phone a friend".
4. When you leave Cabin with Mallory tell her "I wish things were different."
5. When you have sex with Jaye at George's house you need at least 95 love points with Jaye & at least 80 with Mallory.
6. If you've met all that, Jaye will offer you the choice of Mallory only joining a Poly relationship (not a throuple).