Something that both of you seems to not grasp completely ( and it's understandable if you don't have that mindset in real life) is that the MC can say to Annie he had a crush on her for the long time and, at the same time, he can romance Nancy, Dalia and the other girls without finding it strange, because the romantic feelings coexist in his mind without causing him doubt or restraining his actions.
When he's with Annie in that moment only her is important to him, but the moment he's with Nancy or the other girls the same thing happen.
This doesn't mean he's faking love for Annie, or he's compartmentalizing, he just truly love all of them at the same time, and that's what clash with your own ability to read the situation, because you didn't experienced that irl, you can't really accept that completely that's why at a certain point you have issues with it.
We aren't educated to polyamorous relationships, our society is constantly focused on prioritizing the monogamy instead ( I don't want to list all the various influences that concur to create this mindset, religions, families, etc etc, this is not the right place to talk exhaustively of this), so as much as we try to evade it with these games, we always find the MC somewhat of a cheater consciously or not.
Especially when in the story he's not being honest and upfront saying to the girls that he want an open relationship, but to his partial justification Orion at this point of the story is still living in the moment not really thinking to settle it, as a figure of speech the soil is being prepared ( see the jealousy between Dalia and Penelope where Orion is present, Penelope saying to Nova he likes Orion and her realizing she have a crush on him, Dalia and Alex talking about guys they like, I think Nancy already suspect something like it happened with Carla in OIAL and the other hints about it), for when the times of confrontation will happen, but it's still early to see that.
This time Caribdis is writing the story way better than in OIAL imho, he's slowly building the moment where everything will be in day broad daylights and things will be settled ( and also where he need to have the girls in love with him and open to accept sharing him).
About the possibility to visit different servers even without being on a girls path, well that's a bit of a slippery rope, because that would mean to have a more open world where the MC can roam Eternum freely, and honestly I think that would subtract time to the development of both the main story and also romancing the girls ( especially considering already how much big is the game as is), since he don't need to get there to farm money or other objects like it happen on other games to advance the story ( like in NLT games for example).
Of course this is only my point of view, but I think this is what happening to you, it's not simple to really believe in real love when you have it with more that one girl, but this is the paradox with polyamorus/harem games you either accept it as true or else you will have problems accepting it completely.