Do you have a save from before you gave them the wine?
If not if you wait a few days, do they all respawn?
If yes to either, my suggestion is to get Tia to join the MC's party, then go to the camp, and instead of sneaking in, when Tia asks what you are going to do, say "Bash some skulls".
That will make the bandits come to you.
Granted this may mean you have to level up some, and / or advance Tia's quest line some, but it would get around the issue you are having with the one guard.
It's the route I usually take, but can see how that might be an annoying change of plans for you. Though you really can't advance the camp much with out having Tia in your party.