The downside of Fast Mode (of course, not skip mode), is that the pacing of sound, music tempo and some special effects (talking about fast change of images on determined moments) will be wrong. So basically, it reduces percived value of important scenes. But of course, I know some players play without sound nor music (Heathens! xD).For those who are struggling to play this game due to pacing I strongly suggest you turn on fast mode. I really like that this is an actual game and not just a VN but the biggest pain point for me was how damn long everything takes. Turning on fast mode made the world of difference and now plays fairly smoothly.
What I do not recommend is using Skip mode (aka ultra fast mode) very often, (while repeating scenes & stuff),'cause it can lead to bugs and crashes on some systems.
One of these days I'll code that the damn background effect (the smooth movement on images) will work on unscaled time (so don't go faster too). It was not possible to code that before the engine update for 0.21.