I use scene optimizer frequently on characters (including clothing and hair) without issue.
A few basic "rules" that may help-
First, always test it on the subject alone. So try just the character and their skin, for example, not the character in a complete scene. Render it a few times with different lights if you need to. If you're feeling crafty, save this "optimized" subject as a separate scene which you can then later merge into your final scene of other optimized bits.
Second, don't just blindly adjust everything in one shot and hope for the best. I have a general rule of thumb for texture resolution optimization that works very well for me- try to get the final texture resolution close to the output render resolution for key elements (such as characters). So if I'm rendering in full HD and the character has 4k maps I'll reduce them to 1k maps but not below that. I also only optimize any element once, so from 4k to 1k is one shot, not two times of cutting the resolution in half each time although that shouldn't matter.
Finally, don't rely on a script to do all of your work. While scene optimizer is handy it isn't magic, it's just saving you the effort of opening up those files in another tool and reducing them yourself.
Here is an example of an optimized character without any visual flaws caused by the optimization. This scene rendered in 8 minutes and some change on my "OK" but not crazy PC. The biggest optimization that cut a ton of time off the render was to replace the wall textures with a basic black shader- no maps, no effects. The important element of the wall is the window, the window casing and wall material's bump and normal maps were visible originally but unimportant to the scene so they were easily optimized out but not by the script (this also works on lit surfaces, be creative). The laptop, table and chair are barely visible and look fine with a 512 map, normal maps were removed from most of the laptop and all of the chair and table surfaces. The lady now has 1k maps which works and the bulk of the detail comes from the HD mesh and shadows.
Here she is with different lighting (emisive surface from the laptop screen is the only light source) but with the same optimizations to her and her clothes. This one doesn't even have a wall in the scene, just a solid black environment.
These images aren't perfect since I was just playing around with an idea but they are free of any anomalies. Maybe just grab a base character, leave it in the base pose and mess with scene optimizer until you see what you did that caused the anomalies, I haven't had any problems but then I don't do anything really extreme with scene optimizer. If it will help I can probably do some simple side-by-side renders with an explanation of what was done in scene optimizer.
