It is not their fault but they get fucked by association. It is a somewhat common occurrence in real life, from interpersonal relations to business relations. Think of it as a sort of preventive mechanism. Some users (including me) will steer clear of them because it is what we would do in real life (it is just a game but that is part of the immersion). Imagine that it is real life and think of the following.
Consider the House of Ink as part of an urban tribe, where the leader is Reina, she owns the building where they live and also has extended control over their livelihood (she is the investor for their enterprises). You are an outsider that was trampled over by their leader so the logical reaction is to steer clear of them because you have no standing with any of them, not even the bartender (June?). You can see she is reserved during the interview scene, not because she is a bad friend but because it appears her loyalty is with Reina. It is not like any of us like to go around forcing ourselves into random groups that differ from our core values (scratch that, I just described the internet).
Lets pretend that in real life you spent half your life inhaling cat piss (search for toxoplasma gondii behavioral differences) and you are dead set on headbutting your way into the panties of the House of Ink. In real life scenarios to take over a social group, a business or society you have to first eliminate the leader then the competent members and the rabble that is left is the one that you can use. Returning to the fantasy world of the VN (lets not kill them that would be wasteful) you would dominate Reina (make her your bitch) and any of the girls that work directly with Reina and you salvage a relation with any of the girls that are semi independent from Reina (the bartender comes to mind).
So that is pretty much why we would steer clear of them and what would it take to make them a viable option, according to my twisted inner voice.