I'm not sure if it's related to your problem specifically but I experienced something similar myself with the first Honey Select. I had a 1080p monitor at the time (not anymore though) and I wanted to run the game at 2K (1440p), and it always came up with some weird custom resolution instead, it was just a bit above 1080p, but always below 2K. I sort of gave up over time until by mistake one day I realize that my Refresh Rate was set too high for a 2K resolution using DSR (which is what I was using to try and get 2K out of the game).
It turned out that my monitor physically didn't 'support' going to a specific resolution (and beyond it) if the Refresh Rate stayed at 144hz, which was my monitor's default. The thing is the 144hz was default, but it was default for native 1080p only. So my solution was to just reduce the Refresh Rate to 120hz, then set the DSR to 2K (I think it was 1.78x I think?) and boom, it worked. The game launched at 2K just fine. However, once done and exiting the game I'd always have to re-set the refresh rate back to 144hz for my other games and general desktop activity, but that was a minor problem.
Now I'm on a 1440p monitor so I don't have to worry about any of that anymore.