As Evergrey said, you have to keep at it if you wish to get them back. The quest, on success, reads "There is no other way but to keep reattempting this quest in order to get to the bottom of this mystery and rescue your missing slaver..."Even when succeeding the mission a slaver is lost. Now I don't have enough slavers left to do anything so the whole thing just basically wasted my time. This mission is broken.
The game does warn you beforehand with the red skull & crossbones (the only time this doesn't mean dangerous, it usually means pirates... which should be dangerous). If you mouse over said icon, it tells you "Highly dangerous quest." Success doesn't automatically mean no consequences, just things could have been far worse. Either follow the chain through to the end or skip the quest series. If you skip it, it'll cycle back around eventually and you can do it when you have more sheep... er, slavers to spare. Or, if patience and delayed payoff isn't your thing, just hit the ignore option and you'll never have to deal with seeing the quest again.