Need help choosing an engine to work with
Ren'Py is probably the go-to engine for Visual Novels. Straight up story telling, with occasional "programming" elements to shake things up a bit (forking decision making, point-and-click mechanics, basic inventory or mapping systems). It's capable of so much more than that, but probably not at the level you are right now.
RPGMaker is probably the other go-to engine for Visual Novels... but with a lot more exploration focus and possibly basic combat. Anything where STR, CON, SIZ, INT, WIS or CHA type stats are important to your design.
That said, there's a reason why you'll see lots of fan made RPGMaker to RenPy conversions and hardly ever see it the other way around.
[...] a game with extensive and fluid interaction during the sex scenes (2D, animated).
This is the crux of the problem for me. Since I've no clue what you mean.
Some ideas in my head could be done in RenPy with
point-and-click type interactions. But that's not necessarily what you have planned.
When I first read it, my first thought was Unity. I can see why it's been suggested. It's probably the only way you'll retain enough control of what's going on to do anything vaguely on a "fluid interaction" level.
The 2D/animated isn't really a function of the engine and is better dealt with by a 3D renderering program like Daz3D. Create a series of still animated frame that can be stitched together... or create a small looped movie in .MP4, .AVI or .WEBP formats - then simply use the engine to display it/them.
Unfortunately I haven't found anything close to what I've been planning. [...]
And this worries me too.
Again... it screams Unity to me - since almost everything you could ever think of has already been done in RenPy. If you can't find something close... it likely means you have something very, very specific in mind.... RenPy is generic but complex if needed. Unity is complexly generic and hugely specific when it needs to be.
Rpgmaker won't be much help in here. While it won't exactly be a VN, Renpy might work, I'll give it a try. Thanks.
RenPy is definitely more diverse than most people will ever likely use. But if you've already decided that Unity is too complex... maybe the stuff I'm thinking of in RenPy isn't for you either. Again, difficult to know for sure.
My gut feel is that you might need to compromise on your "fluid interaction during the sex scenes".
RenPy would support things like mousing-over or clicking body parts to initiate a new action. But it would be much, much harder to implement something where "rubbing" a body part with the mouse will trigger something happening.
Generally speaking, think "iPad" when considering RenPy. RenPy is designed to work on a number of devices... including handheld tablets (although more android than Apple). If your idea might not work due to a tablet's UI... then RenPy might not be the right tool for the job.