what stone were you missing?, he gives you a key, and you get the cresent moon medallion from the grave, at the ruins it seems there is space anther amulet/key which we don't have yet (I didn't have anyway), which doesn't seem to matter yet, you get the book from the other side.I did that many times, I've only noticed because it seemed obvious I was missing a stone of some kind.
I started a new game, and got it as early as I could, the quest was not there.
I've got no idea which event chased him away though.