My first playthrough was as a fighter, so I've tried a second one as a spellcaster, but I just can't go through with it. Spells having cast times puts them in a position where you can easily be caught by attacks you could've interrupted as a fighter. A fighter can theoretically play solo; a spellcaster is going to get bound up immediately if they play solo. Spellcasters already have the weakness of lower health/health regen in return for fp regen, damage, and versatility, spells having cast times on top of that makes them torturous, and not in a fun way.
Also, I'm pretty sure cast times put them at the mercy of lower attack orders; instant abilities can prempt enemies' attacks and potentially mess them up, while spells with cast times can't.