Youre mixing up what we know as outside observers and readers to what people in-universe know.
Mutants lead to all sorts of problems and innocent casualties, be it by accident, being forced to by shadowy groups or by their own hands.
Take for example that boy wolverine had to kills because he just hit puberty and killed his entire town without knowing what he was doing, all those kids that manifest their powers and end up taking down their families with them, or an apartment block, then you have stuff like Juggernaut showing up andcompletely demoloshing a batallion of the national guard, for the average civilian, they are not gonna know that Cain Marko isnt actually a mutant but the bearer of the bands of Cytorak, they are just gonna see a mutie yet again causing a disaster.
Thats why the mutant registration act had so much popular support even though from the X-men (and ours) point of view its a terrible idea that was a step closer to them all ending in camps.
Also take into account that the mutants we see are the lucky ones that are the creme de la creme, your average mutant is basically the morlocks with pretty shitty mutations that only causes them deformities and harm.
