I'm looking into this right now, you're absolutely right that there's nothing that should justify that kind of performance hit. Any chance you could DM me a save of your problematic game so I could take a look? Debugging is always easier with a test case.
Performance immediately tanks for my system as well. This is noticeable the moment the client loads, it does not require starting a new game/loading a save before my system is making a lot of noise, which is not normal for a ren'py-based game.
I looked at the rpy files and I noticed a lot of loops at initialization, some whiles, for loops, and even some while True at init. I can't easily debug from the client, but I'd guess you have at least one loop not terminating under certain circumstances at initialization, firing more than expected, or some sort of race condition going on.
The game looks interesting, so hopefully performance can be improved.