I see how you got stuck there.

The save file helped a lot.
Basically you studied all three contraptions without saving any captives.
The game basically advanced you to next quest and that is why you think nyx is part of the first quest.
You need to save the captives at the same time as studying the contraption. Basically three visits, three blueprints, three captives. Edit: You know to study the contraption before saving the captive for a perfect three visits only quest completion.
If you have an
older save file, before getting the last (3rd) blueprint, go from there and save 3 captives before studying the last blueprint. Otherwise, restart the game maybe? I have a save file before this quest you can use (page 18 in the saves)
For nyx part (from your save), you have to go to the dungeon yourself (in the tower) and go to the bottom. There you will find the tome of the wisdom after solving the puzzle (riddle answers in the walkthrough.) Then you can craft her disguise and go with her to the burrows. But you can't proceed because when you talk to gwyen she is still waiting for the milking part of the first quest. So that was that. Not a solution but just wanted to mention it.
For tips on how to escape after saving a captive. Basically use a lot of spectral bombs, around 10-15 per attempt. What you need to do is plant a spectral bomb and let them walk into you while you are standing on top of the a spectral bomb. You can walk towards an adjacent wall which will make you stand still while enemy advance a step. Rinse and repeat, and you can exit with using only 10-15 spectral bomb. Note: does not work with normal bombs.
The code for the contraption (left to right) Exchange-Scrying-Sublimation or purple-orange-green. It is the same code for operating the contraption at base. Displayed on the contraption (yellow means orange

)
Good luck and let me know if reloading an old save file or using the save file I shared worked.
The dev has a newer and more polished game called
Space Journey.