Many rpgm games F4 is enough to go fullscreen, but if not then Alt+enter should do it like you mentioned.
Does this happen with every RPGM game?
Are you graphic drivers updated and working?
Do you have multiple displays? And if so are they configured properly?
By the way you are describing things it sounds like you are having trouble running exclusive fullscreen mode, if that is the case you could try running the games in borderless instead, there are mods and apps for that like
You must be registered to see the links
.