You can choose which girls you want to be with. Every combination of girls works. If you want to be with Erika + Martha + Scarlett, it will work the same way that getting Erika + Norah + Martha + Emma + Daphne + Scarlett.
The mini-harems are a way to justify the girls accepting other girls: Scarlett will accept Daphne without hesitation, forming a mini-harem, once she has accepted one girl, merging mini-harems won't be a problem for her, as she is already used to it.
For example, Erika and Norah form a mini-harem of sorts. Erika knows about Norah and accepts sharing the MC with her. Introducing new girls into that mini-harem is easy. If you aren't with Norah, it doesn't prevent Erika to enter the harem, it only forces the MC to convince her in a different way.