My theory on why this seems to happen so often is I think many of the great games that get abandoned were being made by perfectionists. Being like that myself about some things I know just how easy it is to get stuck in your own head about every tiny detail which can lead to no progress being made eventually.
I think there's also a touch of George RR Martin Syndrome.
Basically, a dev starts work only having a vague idea of the plot they want to write. Their plan is to mostly "make it up as they go". But as they go, they start to have new ideas, so they add something here, expand on something else there, drop in a few new characters, and so on. The whole project starts to bloat far beyond their original assumption. Eventually things get out of hand, as they've added too much and are now overwhelmed by the scope of the project. Or they now no longer have an idea of how they're going to pay off all the things they've set up.
Panic and dread sets in, and they start adding in even more new stuff in a desperate attempt to pad out the story and tread water until they can figure out a solution, which just makes everything worse. Ultimately, the pressure gets to be too much, their writer's block gets too strong, and they just sort of shut down. At which point, they abandon the project. Devs with no shame just start up another project (to try and keep milking support or to try and make a better game by avoiding the mistakes they made the first time), but more self-conscious devs may just disappear entirely because it's easier than admitting you screwed up.
