Well... after the brothel scenes, you get to Maronite Village and see a big event there which leaves Sarah a bit traumatized.
Later when you visit Maronite Village again, there are scenes with Yuey, the kid there, and the abducted kid from Aphrodite Village, Dentan I think he's called.
There are some plans to continue the events there, but so far I didn't start working on it.
As far as the game itself, you can finish the game and get an ending by fighting General Achilles near Aphrodite Temple which is through the cave north of the Aphrodite Village Temple.
I'd say you didn't miss much if you gave up near brothel. Well, maybe you did if you gave up early in the brothel.
Did you see the capture of the criminal gang Crescent Moon while doing brothel scenes?
There are also a lot of side events that have nothing to do with the main story that is skippable, like the bandits, Tutsi tribe, etc.