Go for the CPU. 32 GB RAM just (mostly) prevents the game from crashing when loading multiple scenes/looks in the same session. CPU gives you overall better performance. To prevent the game from crashing you can also just load an empty new scene before loading another heavy scene.But having to choose between an upgrade from Ryzen 1600 to 5600 or 16GB to 32Gb RAM, which one would you do first?
This game is heavy CPU dependant. Haven't noticed a big jump from a GTX 1070 to a RTX 3080 for example. If you really want to get everything out of VAM performance wise, you have to experiment with third party resolution upscaling tools anyway. So don't expect a miracle from only upgrading your CPU. Even if you have a NASA PC, the game still slows down pretty heavily with more than 2 people/ lots of long hair stuff. Long hair simulation is the Kryptonite of VAM.