It depends on whether or not you are going to monetize your game. If you're just going to make a game and release it publicly, or through a Patreon subscription, then nothing can really be done to you legally. If you intend to package your game and sell it as a product, then you might run into issues with modders claiming that you stole assets and failed to compensate or credit the original creators. Of course, you might run into issues with selling a product using Illusionsoft assets before it ever reached that point, but let's not get into that now.
Additionally, some of the more thorough modders will include details on when or where they allow/give permission for their mods to be used. But that's really if you want to take the effort to track all that info down.
TLDR: Unless you package and sell your game as a product, you really shouldn't need anyone's permission to use publicly available mods.