Yes he did say that "...hadn't stopped me...", but recall that it was clearly multiple people in there. It's not clear at what point of the "action" the MC starts hearing things since he can't see anything. That situation can easily be understood as Karma "not getting his turn", as opposed to nothing having taken place.How is Karma implying she was raped? He straight up says: "If only the boss hadn't stopped me, I could've enjoyed her that-"
Besides, before Karma mentions the line about his boss, he taunts the MC about him watching the event.
Whenever the MC talks about it with other character, they always leave it ambiguous. I just would like a definite clear answer about it ingame, because that (especially in the context in which it does or doesn't happen) is quite a sensitive topic.