I think the stumbling block is that dialogue skippers (myself included) aren't expecting the dialogue prompt for the book. Especially if you read the walkthrough
the quest itself never states the requirement is now, I forget how the quest item was stated, but it reads like after a few visits. I wasn't expecting to need it after finding them in the garden the first time.
I only figured it out because there nothing else to do in regards to figuring out why I couldn't find them in the garden.
I think we can expect this question asked a lot, especially by dialogue skippers (once again, guilty here).