It's meant to be a bit of a puzzle, and it's hinted at by the various people in the village if you talk to them.To aura-dev:
It should be stated more clearly that portals are linked to villagers, but that the place where the portals send you are linked to the position of the villagers, i.e. it is not strictly a "one villager, one portal, one part of the domain" thing.
But based on how many people are asking about it, I suppose it's too complicated for the average H-game player ^^