Yeah it is a bit tricky but if you check MrBubu pdf walkthrough you can backtrack from the variables presented for it. But in essence in the beggining of chapter 6 after sex with Mallory (in the end of chapter 5) you have 3 choices.
The 2nd one " Let's just play it by ear for now" enables the open relationship for Mallory but in chapter 9 if you also are in an open relationship with Jaye you have to choose between the two of them or you will lose both.
The 3rd choice " I'd like to see what this can become" is basically the best choice for a solo Mallory or the throuple Mallory+Amanda because you enter in an intimate relation with Mallory, if you also do the same with Jaye it will end badly lol.
The tricky part is that choice number 1 " I thought this was about helping you" is basically not enter in a relationship with Mallory, only as a friend but that is the only way for in chapter 9 being in an intimate relation with Jaye for her to propose a polyamorous with Mallory also but it has some requirements including love points threshold.
Gonna do the same i did before and put on a spoiler a screenshot(s) of MrBubu walkthrough so you get it better but I advise you to check the pdf walkthrough.