It would be a scam if the Patreon was solely set up for the game.
Since it's his general Patreon which is also for his art, people who subscribe don't just throw money into the void.
It's easy to create a hate band-wagon for a dev, but at the end of the day, the choice to give money to someone isn't that someone's fault : it's whoever decided to give their money.
If you keep paying Mr Potatoes despite only careing about the game and never looking at the art because you don't care, it's not the artist's fault, it's your own for being stupid with your money (note, I'm using "you" in a general term, not specifically "you, raidenwestler").
In the end, supporting any crowdfunding is a gamble. If you can't accept the possibility that your money might be gone for no gain, don't support anybody. Otherwise, if it happens that a dev is too slow of stops working on a game for reasons and you still keep paying for it, the adult thing to do is accept that you made a poor decision and learn from it.