Since you start by talking about VN, mini-games are uncomfortable, dot.
This doesn't mean that VN cannot have mini-games, but they have to either have a meaning in regard of the story, or/and to be optional. People that will decide to play a VN will either do it to masturbate, or to follow a story. If they wanted a challenge, and so mini-games, they would have choose another kind of game.
Years ago, I said that to define the difficulty of mini-games in VN, you have to lower it until your friend who suck the most at gaming can beat it, then lower it once again. And my opinion haven't really changed on this subject.
It's really frustrating to follow a story for months, then to have to quit because you're stuck face to a mini-game that have nothing to do here and that you can't beat.
This is the worse. Not because of the beat them all button, but because of the trophy. Mini-games should add something to the game, not be the mandatory way to have something big.
Summertime Saga, by example, have few mini-games. Some are mandatory to increase the MC's stats. But they are limited in time, beat them 5 (if my memory don't betray me) times, and you're done, your stat is maxed. And they are related to the stats ; like the workout simulation to increase the strength. In top of that, there's still things that you can do, including lewd content, without them.
The game itself isn't necessarily the best example, its mini-game(s) to earn money not being the best idea. But globally it do it right and it's how mini-games should be implemented in VN(-like).
Oh, and obviously there's another issue that have to be addressed when it come to mini-games. You need to be a good coder if you want to had them. More or less three out of four mini-games are broke in a way or another. And this is what is the most annoying with them.
This being said, it's just my opinion. Do your game the way you want it to be.
It would be hypocritical for me to say that VN should never have mini-games, because I'll probably to put few in my own game. But I'll do it following what I said ; not too difficult (or with different level of difficulty), relevant to the story, limited in time and not "absolutely totally at 100% mandatory" to advance in the game. The goal being to add something to the game, but a something that will not be annoyance or frustration.
Oh yeah, it's added for a reason, to hide the lack of imagination of the author and the lack of content of the game ; add some grind and the 5 minutes worth of content you added will take a full hour to be seen.