Well...I am curious who in their right mind thinks that a person can react fast enough to play the brawler mini game. There isnt enough time to even pull off the simplest of arrow combos before some hidden timer runs out.
... this seems pretty much like a skill issue? Like, i won't pretend it is easy or anything, you're certainly right there's very little time with the starting stats. But it's evidently not impossible, and after you win a fight with the weakest opponent and put a point in Speed it gets considerably easier, and with 3 points in speed (after you beat the middle guy in the lineup) it gets downright comfortable.