For me, at least, it mostly just matters that I know the character. If it's a character from a setting where I haven't interacted with the source material, it feels like I'm missing all the in-jokes (I probably am!) and that grates on me.
I've never watched Naruto, or One Piece, or Dragonball. So anything from those three shows is an automatic no from me. It doesn't help that I find the designs for the girls in those series to be rather uninspired.