Yeah for the fire staff, you should have it on your quickbar as an opener basically and be prepared to quickly swap back to the frost staff. It can be used occasionally, it's just too unreliable as a main weapon. The frost offhand was pretty much always better than the fire offhand in my experience. So you don't need to worry about swapping that. I never enjoyed using the fire offhand. If an enemy is threatening you, better to have an instant "Oh fuck" button from the frost offhand aoe than have to target specific tiles with a cast time that can be much more easily interrupted.
The real threat as a mage are the enemies that can jump around you (Wolfs, fish people, bats, etc), and the fire offhand and basically any attack but the frost aoe becomes unreliable or unusable against those enemies. Frost offhand is basically a requirement as a mage (As is all points in wisdom so the frost aoe actually does damage).