I feel a lot of people are forgetting the Novel part of Visual Novel. Also, events take time to build up to and prepare. Yes the Real World isn't as glamorous as the Magical World, but I think important things are being set up at the moment, in both worlds.
In the Real World we have to try to find our birth mother and get answers from her. But that aside, we're actively expanding our harem with gorgeous girls.
In the Magic World, cracks are starting to show in an important characters mask. Other important events will almost certainly happen in our absence from that world, events that might only be able to happen in our absence.
Also, I strongly suspect, that things are going to progress to the point where once we've recruited various girls into our harem, and gotten what we need from the Real World, that we'll probably arrange to forsake the Real World and move, with our girls, to the Magical World permanently. And from there the story will progress from wherever it is at that point. Part of what's caused this "problem" of the Real World is how MC was really missing his family. Having sister, and probably our mom too, with us in the Magic World would remove that issue for MC and have no real reason to ever return.