Am I doing something wrong? I'm on the second floor and I'm just wandering around completely lost. For combat I'm only using my default attack because I have no skills and the few I've found don't do any more damage. Constantly having to leave due to fatigue and poison slows my progression even further.
You can get skills and passive abilities by looting everything you see. Pots, Chests, Crates, etc
Then you equip them! Bear in mind, some skills only work for some weapon types.
Pay attention also to what your weapon scales with. Daggers and shit get a bonus from higher SKILL. Big ass swords of course get bonuses from STR.
This isn't a game you can necessarily rush, you may need to explore more and acquire more ether, loot and skills before you can rise up to later challenges.
tl;dr get to grinding