The main issue I have with shrink houses is an inconsistent ability to jump from box to box.. Use to be that some keys spawned way too high in older releases, but I haven't tested it in the new release as much. I don't hate the shrink houses, but they have been a definite avoid for me since I've had such a bad taste in my mouth from other releases.
It felt like sometimes the giantess would give you like 10 seconds to complete the challenge and other times maybe a minute or so. This inconsistency of jump height, ability to get a key or not, jump distance, then time to complete challenge is what I think it's true downfalls are. It's a cool concept to add a fun little platformer, but consistency is key here, not that key location has to remain consistent, just mechanics. The book elevators are a freaking annoyance to jump onto miss a book and you feel like your world is ending, gotta say that hurts when using them half the time, the book position is sometimes really hard to account for since it is in between boxes, that overall mechanic IMO is just painful. I don't mind elevators, it just doesn't feel right.