I'd recommend
City of Broken Dreamers, honestly the best 3DCG visual novel I've seen so far.
Utawarerumono really stands out among the older games. It's a hybrid VN/turn-based strategy game, and the story and setting are frankly amazing. There's several sequels, though only the original game has sexual content.
Fate/Stay Night is another obvious one. This is where the Fate series started, and it's worth checking out even if you don't really care for the rest of the Fate series.
Two 90s titles probably worth checking out are
YU-NO and
Divi Dead. In both titles, you have to do your own detective work to unravel an overarching mystery.
Speaking of mystery;
YOU and ME and HER is... not the sappy highschool romance VN that it initially seems.
And as far as RPG Maker games go: anything by Leaf Geometry is guaranteed to have a ton of worldbuilding. The SEQUEL series and
ASYLUM are set in the same universe. If you don't feel like doing the SEQUEL series (four games, each of them huuuuge), then just try ASYLUM. That one, especially, is atmospheric and trippy. Especially if you're going in blind.
EDIT: how could I forget? Check
Black Souls and
Black Souls II. The visuals are rough around the edges, but the gameplay and writing are simply sublime.
Especially in Black Souls II.
Also try
The Heart of Darkness, which is a pretty solid Black Souls clone with much better visuals.
