Some things that don't have written out HP can still be destroyed. But somehow recover after some time passes.Everything that says that has HP can be damaged so your walls can get destroyed, you can cheese it with stairs from what I understand. Anything involving placing stuff is cluncky overall
At least, I saw that happen to my chicken coops.
I think there is wisdom in understanding the game's pathfinding and using walls to goad invaders into fields of spikes which finally lead to your defenders. With how bad pathfinding is, you'll probably need at least one gate for every direction for this to work.