It does matter if you need the valour gems. Getting 1 from goblins instead of 3 from orcs is pretty significant.I just leave the castle defense alone bc it just doesn't really matter what setting you choose.
I've been attacked by goblins, bandits, and orcs that were repelleled. Then a random group of undead decide to attack for no reason despite the fact that my castle defense is on easy/medium