I suspect that you're using a preset for the environment map and that preset is probably a scene instead of just a hdri, so to just load the hdri:
First load the scene where you want to use the hdri, then go to the environment tab, click on the small image (first screen), then click on browse (second screen) and look for the hdri (the file location is also on the first screen) -> double click to load the hdri, done.