You sure Goblins attacked? For me, when I change difficulty to medium, attacks still are random. I always save before entering castle and sometimes I'm being attacked by goblins other times it's undead. Saving before entering castle is the best optiom in my opinionI think there's a bug in my playthrough.
When I set my Castle Defense on Easy, I was repeatedly attacked by Goblin with the strength of 38 when the limit should have been 15.