If you squint there's some justification for this already, I started a new save when I came back to it recently and IIRC the initial slave quest mentions a very long waiting time for licenses for the majority of people, after making the initial 5k payment. And then the NPCs who do have slaves mostly seem to stop at just a few, or just one, despite being wealthy.
If the game mechanics had any actual downsides or added expenses to stuffing 100 people in overcrowded rooms in your aunt's giant, otherwise-unused house besides bloating up the code, it might make more sense that people generally don't do that sort of thing. But of course it's not that deep of a management game and if it were that might also cause development to outlast the heat death of the universe.
"Outlawed modifiers" does seem like it should be a thing though, unless the Angel Nectar companies are in on it.
