I wouldn't worry that much; admittedly I'm still a reasonable small fish, but out of ~400 patreons, less than 5% are regularly engaged in giving feedback, and in 90% of the cases they're supporting whatever vision you've got... that's why they're there in the first place.
Just be clear on what and why you are making the decisions you are, take feedback into consideration (often they have good ideas, or have spotted some plot-hole you'd ignored, or have balancing feedback that comes from being a player rather than a developer), and always remember that the reason they're there in the first place because they believe in you. It's only when you squander that trust that your relationship with them turns from friendly into antagonistic.
I agree with the general sentiment of this, but I have watched a number of projects go south because some vocal and opinionated minority of the patrons started organizing shit against the devs. It is hard to quantify interference from non-patrons, but I am 100% certain that there are actually a lot of adult games and media projects which are under assault by outside provocateurs pretty much at all times.
When a project gets "big enough" it can attract the attention of saboteurs that are ideologically motivated to end the project by pretty much any means necessary. This often gets written off as "internet drama", but it actually is within the context of the ongoing vice wars by the agglomerated regressives, conservatives, and rightwingers of the world along with certain puritanical authoritarians in the center-left and corporate political coalitions. This can and does result in things like FOSTA/SESTA being enacted in national law or the Comic Code or the Hayes Code or the ESRB or Motion picture content rating system.
All of this is a reality and risk that people need to understand going into any crowdsourced or crowdfunded project. There will be people who will pay real money to sabotage your efforts if they believe that is their purpose whether God, king, or ideology. And they may well pay money to you to get access to your community. And being nice to them won't necessarily win them over to not attacking you and your project let alone actually supporting your efforts; I don't mean just one impotent person either because some of these people are in the top 10% of consumers in the world or are part of royal families with trillions of dollars collectively.
All of this creation and production does not happen in a vacuum; it is deeply political even if the devs want to espouse political neutrality or apoliticalness.
This stuff doesn't matter at all for the vast majority of projects. You have a cute cis-het couple fucking? Not likely to attract ire. You have a cis-het couple engaged in adultery? More likely to attract ire. The further you get from the common "proper" patriarchal norms of human sexuality the greater the risk becomes that you attract the ire of fundamentalists. Particularly if you get popular or socially influential.
People have very strong opinions on the amount of control they feel they should exert over a developer's vision if they're pledging to a game. As a budding dev group, for us this is both illuminating and horrifying. It's incredibly difficult to produce any game, let alone something other people will value enough to play. Things become even more difficult when one adds patronage to the equation... and then on top of it there's always the expectation and pressure to modify one's vision on the fly to suit patrons' wishes, all while keeping up a steady output of quality content.
This isn't meant to be an indictment of anyone's preferences or wishes, but merely an observation; it's obvious why so many amateur porn games end up abandoned, or flat-out failing. There's just an ungodly amount of pins to juggle at the same time... And that's all without getting into more esoteric questions like the quality of the game content itself.
I have a pretty strict policy of non-interference when it comes to my patronage. I quietly push dollars in the direction of devs I support, and I simply pass over the others generally in silence. The projects I adore I will send play test reports in and do bug reports on.
What I stipulate here is just to inform about my policy; I don't like making normative statements or demands. I know from first hand experience how damaging to creative process that is. If you want to get my dollars and my labor then there are some conditions your project has to meet. If the project doesn't meet those conditions or does but ceases to as some point in it then I withdraw my support; I might give warning before I do so, but often, it will be quiet by my standards, and if I am giving warning then the people who don't provide feedback are likely ghosting in higher numbers.
there's always the expectation and pressure to modify one's vision on the fly to suit patrons' wishes, all while keeping up a steady output of quality content.
This often wrecks projects. The more you compromise the vision of a game and particularly the process of development to bend to the will of a vocal minority the more you risk the entire project. There is wisdom in listening to and adapting feedback in service of the vision and the process, but appeal to short term material support by vocal patrons does not necessarily translate to sustainability, profitability, or general support.
This is where a lot of projects get lost in the weeds. They have some number of initial followers that are often passionate or at least very vocal. The immediate success of the project can often depend on these people spreading the word and recruiting new patrons. Supporters from the beginning of the project can get very possessive about it and express entitlement about it. When devs cater to their whims over the interests of future supporters and players, they can cut themselves off from more general communities resulting in a progressively more stunted project.
You see this a lot in the adult game dev community. Where a proof-of-concept will be turned into a production model at the urging of the initial supporters which backs the devs into an untenable and unsatisfiable condition where they need to abandon the tech demo that got them the initial support in favor of migrating to engines better suited to their needs or to accommodate artists and other devs that have been brought on. This often results in a game which is built on layers and layers of successive mistakes that can't be fundamentally reconciled without doing something that would set the initial supporters on equal footing with new and future supporters and risk alienating them. Especially when the initial content promises are for things which can not be marketed like sexual child abuse imagery, desecration of corpses, or other things which are illegal or criminalized within the dev's jurisdiction or the jurisdiction of Patreon or Paypal or credit processors or publishers or any of the many intellectual property right agreements the devs are bound by.
The community at some point must reckon with notions of ownership, entitlement, and notions of consumer rights. Patronage is not stockownership of a corporation; patrons do not own the rights of the production or products. Patrons are not buying the product as they would do on say Steam for instance. There is a distinct difference between handing the Steam market 60$ for a game versus spending 60$ on a patreon project. There's an even larger difference between that and handing a corporation like Microsoft or EA 60$ in exchange for some portion of ownership of the company and voting rights on the board of directors.
Patrons must remember that they are not buying their way onto the dev team; the patrons are not buying employees and they are not managers, and the dev teams need to be very careful to be explicitly clear about these boundaries with themselves and with their patrons and beyond.