Switching gems is interesting , but does not seem like something you should be able to do in combatYou can change gems before the first card is played in combat. As we get more advanced enemies, and enemies with immunities to things like poison, having the ability to swap between hands will become useful.
As a poker addict the idea of advanced enemies sounds good .... Variation to keep people on their toes and not just doing the same old thing ! The one idea I can think of from comments here is a weak enemy who reflects a percentage of damage back . So if you go for overkill all the time you get hurt back !