IMO like half the joy of making characters is putting them through hell. The more you love a character, the more you'll inevitably drawn to making them suffer. I honestly think that's one of the most normal things about Sel, actually. If anything, I think he's pretty gentle. Maya's so broken the worst of the worst doesn't really affect her anymore (she's basically only scared of losing sensei as far as I can tell), Ayane is strong willed and adaptive enough to bounce back given enough time and sensei, Rin has a tight nit support network, and so on.
Actually, thinking about it I think the character he's meanest to is Kirin lol. Where most other characters goes through much worse then her, they also have something that helps them keep going, be it personal resilience or friends they can rely on. Kirin meanwhile has Miku and maybe Makoto as of recent events, but I think her personality keeps her from even trying to rely on others. I think the way every problem builds up against her without a real outlet is probably a good amount of what makes her the way she is. Maybe she'll chill out if she can get to the point where she can actually talk to Noriko again