Sorry about that. It's kind of may nature. I too, am a bit of a dick. Well that's not accurate. I'm a complete asshole.
Well, that's okay. I apparently misjudged you if you're capable of that kind of self-reflection, so I apologize. Let me give you a more detailed explanation of why the system has to be the way it is right now:
Take the scene in which you're introduced to Erato as an example. She approaches you the first time you leave town after learning about the hunting mechanic. By that point you have already left town several times without meeting her, establishing that location without filling it with content. If I didn't force the player to manually leave town before bringing up the map, the whole Erato introduction would become much trickier and wouldn't establish a location in which the player can reliably go to visit her in the future, and I'd likely have to interrupt an action selected by the player and do something unwanted with it instead. By forcing the player to walk through a very limited number of locations, I can make sure he sees that scene before encountering any other wilderness scenes after learning of hunting, which makes things much simpler and more predictable.
A similar situation, but not related to the map, happens when you first see Samarra and Nell riding into town. Because the game forces you to visit the town itself before you can get to any other locations, I can guarantee that the player sees their scene if he leaves his house that day (or after) before he can come across any other content. That reduces the risk of creating content that happens out of order.
That's why travel will always be centered on two locations: The town location and the wilderness right outside of town. You have to enter one to get to the other, and each provides access to a separate map. It allows me some degree of control over the narrative flow in a nonlinear game.
That, and Ren'py is a bitch when it comes to dynamic actions though GUI elements. The current map button is a nasty hack, and it would be even worse if it could be used from more locations. Had I known what I know today about the limitations of Ren'py, I would probably have designed the core mechanics of LLtP slightly differently (perhaps in a style closer to a game like Summertime Saga, a game that has had its own fair share of game-breaking bugs related to its map icon), but it is what it is and I do what I can with it.