Yeah, pretty sure it's explicit canon that giantesses cannot be shrunken by anything, ever. Normals cannot be grown. Some giantesses can grow themselves, it requires a large amount of magical power though. For most giantesses, growing is a temporary deal, due to the aforementioned large amount of magic it takes, the exception being the queen and likely Croixelle, given their absurd amounts of arcane reserves- though in the queen's case, she can grow even larger, and that might be a temporary deal- it could also just be that if she were to be at her true maximum size, she'd just be too big to interact with normals at all, let alone have the castle conceivably accommodate her.
You know, that's kind of a missed opportunity tbh- I get it's a porn game, not a scenery porn game, but it would be really nice to see landscape shots of the various kingdoms' main cities. When you think about it, the castle of Timfalme must be obscenely massive, like an architectural mountain rising from the city, casting a massive shadow as the sun crosses overhead.