The problem is that I go from 0 fatigue to 24 after a single fight on level 3.
That was my experience as well. I've managed to clear the first two rooms in a single day, but every room after this was 1-room-per-day kinda deal, with the exception of the boss fight, where I took the risk of fighting with 26 fatigue and won. I was kinda surprised since the fight wasn't all that difficult, even with me wasting some turns before realizing the gimmick.
I was playing on Warden, with a build that I've intended to be Mind+AGI, but by the end of floor 3 was more like AGI+STR, and finally ended up as AGI+Charm, because of the strip club, which I had to use a lot, due to some poor planning and non existing income. Took me 34 game days, but I've managed to get the Empress ending as a complete virgin.
One thing that playthrough taught me, which I'll be sure to make use of going forward, was the value of saving Order boosting edicts for emergencies. While the utility of saving yourself from reaching 0 Order is rather obvious, I can't stress how big of a difference it was to fight some tougher rooms at, say, 60 vs 40 Order.