[Note: I'm not a lawyer, just an informed member. Therefore this should
not be seen as legal advice, but just as shared knowledge]
If I were to unpack character models and meshes form a game, say mass effect and use them to make renders to make my own VN game, how would that work with copyright?
That would works really badly.
Using screenshot of a game could possibly be presented as a tribute, or the desire to create a fan fiction. But what you intent to do have absolutely no way to be seen otherwise than an intellectual properties theft.
In top of that, accessing those models is part of reverse engineering, what is illegal in many countries ; you take the software as it is, and don't have the right to access what is stored in the files. Therefore just the fact that you gained access to those models is by itself illegal.
This being said, the reaction will totally depend of the rights owners. Some will not care at all, and therefore not try to enforce their rights. Others will not let you go until they have the guaranty that you'll never ever try to do this again.
Seeing that there are many games here about real and popular franchises like Avatar and DC and there seem to be no problem.
The absence of problems do not imply that what is done is legal. Unless explicitly stated otherwise, you've to assume that those games exist because they haven't drawn the attention of the rights owners. Whatever because they stayed unnoticed, or because they are so underground that they don't worth the time, and money, needed by an action.
How would that work if I used real assets from the gamefiles and make renders with it (modifying them for obvious reasons ofc)?
Well, someone at EA will see your game and find the models really similar to the ones used in Mass Effect. He'll notice the legal department, that will ask someone to investigate.
The investigation will conclude that images do not come from the game, and that there's no models this similar to theirs that can be found. You clearly used their meshes, that you tweaked by hope that you'll then avoid any problems.
You'll receive an email asking you to stop this immediately, or to be ready to face the consequences. If they are in a bad day, the email will just tell you that you'll face the consequences, period.
Of course, to win their lawsuit, they need to prove that you effectively used their assets. But since you'll probably have to face US justice, there's way to goes around this problem. Presenting you like a bad guy (hey, you frequent a forum full of pirates !) could be enough ; if you do this, what is proved, you probably also did what can't be proved.
But in fact they don't care to win the lawsuit. The judgment will not come before years, and in between everyone will be afraid to use their franchise in a bad way. It will cost them money, but they can totally afford this, they've lawyers payed yearly just for that ; for once they'll be payed to do something.
As for you, it would cost you so much that in ten years you'll still be paying for this. They totally know that, and totally count on it for you to beg them for an agreement.
Or you can be totally safe because they'll decide that you don't even worth that they think about you and your game.
It's the most likely outcome, but it shouldn't make you forget that it can end differently, and that you would really dislike this difference.