Lena is able to walk in on Holly In flagrante delicto a few times before that point in the game, and also notes how Holly is purposefully trying to be like Ivy.The lack of any evidence, and how it undermines these theories when they concern Ivy instead of another character is precisely the point.
Lena potentially never seen Holly fuck, period, and yet her remark is taken as a gospel. As if she is suddenly an expert on both Holly's sex life *and* drugs, when she has potentially never ever taken any herself.
To be exact, she offers drugs to Lena once. If Lena turns down the offer Ivy herself concludes the next time Lena doesn't want any, and you have to make an active choice there to get drugs from her.
Given this, the presumption that Ivy is drugging Holly against Holly's consent is not just far-reaching but actively ignoring/contradicting what the game actually shows us, regarding Ivy and drugs.
But there's not a lack of evidence in Ivy's case. Hard evidence sure, but to completely dismiss it out of hand as preposterous (especially since that was a big plot line in EK's previous game) is just denial.