cant find the correct formula for abigail's compond/experiment anyone mind telling me what it is
It's different each playthrough. I can tell you a strategy though:
On the first two attempts, choose vials 1-4 and 5-8, respectively. This tells you which vials are the right ones, and might even give you the position of one or two. Even if none of the vials appear in the right slot, you can deduce the right order over the following attempts, so by the fifth and final one you know where each vial has to go. It requires some logical deduction, but after a few times it becomes rather straightforward.
Try different permutations of the correct vials, and use logic to which permutations are possible. If everything else fails and you simply cannot get it to work, you can create a save at the beginning of the quest, reload as necessary and try different combinations and permutations of vials.
Hint: You can look up your current and previous attempts in the notes next to the lab setup. No need to write down everything yourself. Also, for me it helps to just use their numbers (1-8) instead of their colors.
You can even do this with a handicap, namely when you want a particular vial to go into the first petri dish. (Lab accidents can happen, and the results depend on the color that was in the first petri dish during the _previous_ attempt.)
So even when you use the same vial for the first dish every time except for the final attempt, by then you can reliably figure out the correct combination.