You need to start a different quest to be able to visit the estate, a quest about a scholar in Valos looking for rare books, he should be in a house to the east of the Heatherly Estate if memory doesn't fail me.
At some point his quest will lead you to the Heatherly Estate to try to get a copy of the book. With high Intrigue you can potentially gett he book just by dealing with the guard, but you don't want to pick this option, instead you should agree to meet the owner of the manor about the book, this should start the "Domestic Dispute" quest.