This game is harder than any other game the dev made imo.
IMO Sequel Blight is a bit harder, mainly because that one is less streamlined.
Asylum is doable enough if you know how to use Aryaum's Jewel. Aryaum's Jewel boosts all your stats a little bit except for Luck - but if you go to the Mirage Grotto, this stat bonus increases with one point in every stat for every floor of the grotto that you complete. This bonus is permanent, and AFAIK it doesn't have a limit. So you can raise your stats indefinitely as long as you keep grinding in the grotto.
There's also the Left Eye, Right Eye and Third Eye miragebrands, which are just broken. You need to equip all three of them at once, but they'll give you a 50% bonus to all stats (even Luck!) + immunity to all basic ailments.
...but if you don't use that stuff, then yeah - things are going to get very very hard once you get to the reawakened Distortions.