Short answer: Physique will boost your physical damage, crit damage, and base unarmed damage. Higher physique benefits unarmed weapons (dusters and demon stones) more than swords or other melee weapons, due to increasing your base
unarmed damage.
Longer answer: A funny thing happened on the way to answering your question. I seem to have discovered a bug in the game, not sure if it is known or not though. So, take a look at the pic below:
View attachment 2254198
With 91 Phys, my damage is actually lower than my damage at 83 Phys. After eyeballing my stats between the two ratings, I noticed that at 91 Phys, the bonus physical damage and crit power are NOT applied to my character (base unarmed damage is registered correctly). But at 83 the physical and crit bonus stats ARE applied.
Edit: 100 Phys applies stats correctly, but 90-99 doesn't. Not a game breaking thing, but weird.
High physique is good for unarmed weapons, Dusters and Demon Stones, because you base unarmed damage is 20% of your Physique stat, so not only do you gain more Physical damage and Crit power, you also increase your BASE damage (before modifiers).
Normal melee weapons gain the benefits of Physical damage and Crit power boosts, from Physique, but their base damage is not affected by Physique (as far as I am aware), and the difference between 70 Phys and 90 Phys, isn't that great of a boost, so having a really high Phys with a melee weapon (swords, knives, maces) isn't as useful as it it for unarmed weapons.