But this is the same kind of deflection I've seen from others. Bringing up the broader historical context, but not clarifying what the accusation actually means. Why am I 'fetishizing' the transwoman, but not the cisman? Does the term 'fetishize' have any meaning beyond spiteful moralizing?
In general, the word 'fetish' means an inanimate object that is believed to be inhabited by a spirit or deity, and as such, used for worship or reverence of said spirit or deity. Sometimes worship includes real or imitated sexual acts, and I think that this is how the term came to be associated with sex in general. A sexual fetish is an object or activity that creates sexual arousal. One can say that foot fetishists "worship" feet (although it all differs on a case by case basis and so on).
Sexual fetishism is a bit tricky. Some consider it a sexual deviance, but it's not quite so simple. Most people have some kind of sexual fetish, so it's actually normal. The difference between norm and deviance is in how far one takes it. For example, a person might fetishize animals and use animal costumes and accessories for sexual activities (i.e. roleplay). This would be regarded as a normal fetish. But if they were to have sex with an actual animal, that would be a deviance. Of course, these are very basic examples, there are some specific cases, like if a person wants to have only animal roleplay sex and forces their partner into it, then it's a deviance as well.
Now in case of transwomen, way I see it, the people accusing you of fetishizing imply that you're only turned on\ want to have sex with a fully transitioned transwoman (or almost fully, since dicks are involved), and not with one who's barely begun the transition or is halfway through the process (i.e. one that still looks like a man), or even have a normal relationship with them. And here it would depend on your actual preferences and how willing you are to discuss these things with the accusing person. The absurdity of this is that trans people themselves would more rather be fully transitioned, although it, again, depends on the individual person. We should also not rule out the fact that regardless of transition stage, a trans person still has needs, both physical and emotional, so even if they'd rather be fully transitioned to feel comfortable with themselves and those around them, they'd still appreciate someone to love and be loved by while they're still getting there. So, you know, all is relative.
As to whether or not one can fetishize a non-trans man or woman, it's actually very much possible. It's just that it's normally accepted for men to be attracted to women and vice-versa, so the lines are a bit blurred here. Likewise for gay people, but since they too have a history of being marginalized, nobody says anything to them in this regard. Again, the bottom line is the degree of fetishizing. If you are attracted to a woman and see her as a human being of the opposite sex that you're willing sleep with, then it's fine. If you see her as some kind of an idealized sex toy, then it's not fine.
The key word is 'objectification', I guess.
Still, keep in mind that the world is full of different people, some of whom happen to be pretentious assholes who like to project their own problems onto others to feel better about themselves. Some are hurt, some are stupid, some are dishonest with themselves. So if you're willing to have a normal discussion with someone who accuses you of something, but they keep accusing you regardless, then they either have personal issues, or are following some kind of an agenda. In either case, they're not worth bothering with.