So I checked it and the game indeed needs EXACTLY 5 girls for the quest trigger of getting 5 girls to be activated.
As such if you get the mother and daughter pair last then it goes from 4 to 6 girls and does not activate the trigger.
To solve that you should save the game and then use the save file editor.
Go to advanced options.
Open the tab "variables" and edit the "slave count" from 6 to 5.
Then save the file with the editor.
Afterwards load the save in the game.
You will get the info appearing that you got 5 harem girls.
That means your quest trigger is activated.
Save the file again after that.
Open the save file with the editor again and change the "slave count" back to 6.
Save it again.
And finally load the save in the game for the last time.
Now you can recruit the goblins and have the correct "slave count = 6" in your playthrough.