So I figured out the issue, thanks for all the help. If I started out anywhere but in the bathroom before evening I would be locked out, I had to transition into the bathroom at midday and then fast forward time until evening to be allowed in the bathroom with Lyriel. Weird. Again, thanks for your help though.