Yeah, a dungeon-crawler isn't a VN, because it has gameplay.
I'm thinking the problem you'd have with a first-person VN is the "walking simulator" syndrome. It's the same kind of complaint you get when a VN is made in an engine like RPG Maker: the player ends up spending most of their time walking around looking for NPCs to interact with. Well, it'd be easy to fall into that same trap with a first-person real-time 3D kind of game (or 2D for that matter), unless the amount of walking around was limited or carefully created to be exploration. You could teleport the player between interaction areas with a world map so that the area they'd be walking around in would be to a degree limited, but then you're kinda losing out on some of what the genre gives you.
The core of the VN is really the story, and the interaction kinda layers on top of it to different levels of game-ness. You might just have some options to choose from, or you might have some combat of some kind, or minigames, or a world map, or a day/night or real-time-like system etc. etc.
You've got your straight-up-VNs-with-RPG-like-battles (like Monmusu Quest), your pretty-much-RPGs-with-lots-of-character-interaction-that-sorta-operates-like-a-VN (like Lilitales, which I might've spelled wrong), and your actually-an-RPG dungeon crawlers (like Prizna or Princess of Puppets, or whatever its name is) for some examples of where the line between them sorta blurs. It's kinda like how the line between a real game and a VN blurs when you start to add game-like elements to it.
I guess it's rare because it's not straightforward to take advantage of what makes the first-person shooter/walker/whatever sort of genre distinctive. You see the occasional Japanese third-person adventure VN, where there's maybe some item hunting or other quasi-gaming elements. The West's fledgling porn games scene, though, seems to rely pretty heavily on game development engines even now, which might explain why rare game types feel so rare.