Not... entirely true. There is one instance where your Charm rebounds on you, when you first meet the Asian girl at the Aquarium. You try to Charm her, and it rebounds onto you and since it's your own Charm spell reflecting back onto you, you can't Mind Control yourself into permanent arousal, so you spend the initial Charm duration super horny and fucking the other Aquarium babe (the one who can safely carry the Dragon Gem).
Other than that one-time event, due to how Charms are complete ownership of the target, once any hostile Charm lands on the player, unless you have defense items or training (with sufficient mana) it's game over your mind now belongs to the caster.
Dev isn't against adding further characters though, so if you really wanted to, you could come up with the backstory to another character and request it be added in that is capable of longer-term reflections of your Charm spell, that won't result in Bad Ends.