Maybe, though I think it wouldn't have been abandoned so quickly after Orlov's death and a sudden silence from the rest of the castle's inhabitants might well have been noticed. And there isn't an obvious path to the castle, I don't think it would have faded that quickly.Castle could have belonged to Count Orlov & Master just moved in after his escape?
Perhaps Orlov's seat was on the path going south from the Ruined Tower? And his wealth derived from being the main trading point with the dwarven city you suggested a while back?