You're talking about workers, not slaves. If I'm trying to play the game as a slaver there is never a point at which "slaves probably aren't an issue any more."The prisoner rate in forts seems higher, but by the time you can take on a fort, slaves probably aren't an issue any more.