It seems you can get soft-locked into not being able to leave fairhaven if your hips grow and you clothes dont fit anymore. Either that or I encountered a bug that deleted all the clothes I had bought. I went and reloaded a save and they were there, so I think its the hips.