Yeah, no, that's not really how that works. Think about it more like if you said "we're going to the Miller's" (meaning the house of the Miller family.)The apostrophe is fine but adding "the" to the sentence implies there are multiple restaurants belonging to the Berrie family. It would be like me saying I'm going to "the McDonalds". Adding "the" implies I might be going to multiple locations.