What kind of controller are you using? I know, if you're running a version of Windows, it will find some drivers automatically, but do you have the right controller drivers installed? I've run a controller with these games fine personally, and I'm using a bluetooth Xbox One controller.
Have you checked for stick drift via
Set up USB Game Controllers > click your controller > Properties to see if anything is giving slight drift? The game doesn't have modern "deadzoning" so any drift will be interpreted as input.
For reference, this is what my current controller looks like in said menu.
View attachment 4414869
If you do have drift, you can calibrate the controller on the settings tab.