AI Video Generation from Images - Open Source

[Xavster]

With my current game Stellar Crossroads, I am planning on creating a music video utilising in game characters. Originally I was planning on creating a series of 50 or so images and then using various transitions to match them up with the source music. Basically a slideshow style that you have likely seen in other games. The music video will be roughly 3 minutes long and I will stitch images / videos together using VSDC (Video editor).

As an idea I was wondering if AI could be employed to animate provided static images to provide a series of videos that I could stitch together instead. I would need to animate the actions of lead singer, lead guitar, keyboards and drummer. Are there any recommendations on an open source AI that may be able to do this?

 

[osanaiko]

As far as I know, at the moment Tencent's Hunyuan is the only open source AI model that is reasonable capably of generating video in a hobbyist environment - by which I mean you can just barely feasibly run it on high end gaming GPUs (3090s etc). If you are enthusiastic enough you can also spend time and money to learn how to use the tools in a cloud instance and rent a higher end GPU like a A100 for a few hours, which will dramatically improve your productivity.

I don't know how much "disney animation" or "3d generated images" it has been fed as input - the output i have seen appear to be mainly based on movie or tv show style imagery.

Unlike the image diffusion tools, I don't think there is any LORA or finetune capability yet, so you can't get specific styles of output.

Apart from that, there have been various static images->GIF plugins for stable diffusion et al, but those don't work on a video sequence so you get constant flapping of details between frames.

As for commercial offerings, I've briefly played with "Lumalabs" dreammachine free credits to generate a couple of very short vids, e.g. https://f95zone.to/threads/the-moor...ish-girl-slow-corruption.205564/post-15705519 . The newest version of dreammachine lets you set multiple "keyframe" images - perhaps that might work for your sequences? unfortunately it costs money to do more than a few generations per month.

 

[Xavster]

To clarify requirements, it's not so much open source as free and relatively easy to use. Given that I have fairly modest requirements for it, I don't want to invest too much. Really what I am seeking is something that can turn a static image (rendered by me) into say a 10 second clip (by AI) based upon a description.

Bit like:
Feed in image of a girl playing drums with prompt like "plays drums aggressively."

If it can create a video, I can stitch it together with others to create the full video. I really don't know what the tech is currently capable of and thus don't want to research for a week, only to find it's a dead end.

 

[osanaiko]

Revolting and mentally unwell it may be, 4chan/b/ is the place: it seems that the AI video threads on that board are showing off some of the most cutting edge amateur AI video work for Adult theme stuff. Lurking and following the occasional links that get posted to info sites / tools might be educational.

However as I said above, there are companies with for-money tools. They are also the ones to heavily restrict the content to be Safe For Work. In your case, a rock band scenario would probably be okay.

Regarding "not wanting to try it unless you are reasonably sure of a good outcome..." all I can say is that there is no clear answer for you. It's a very immature technology, and just like Stable Diffusion, the people who get good output are those who spend a bunch of time learning and iterating. No path forward except getting your hands dirty.