Generating AI images with StableDiffusion - Beginner Guide | and with img2vid

[neenerpants]

I am by no means an expert in this but generating consistent characters is not easy but doable. You can achieve somewhat consistent characters by utilizing different prompting techniques and by describing the hell out of how your character looks in the prompt. the more you describe the less the AI has to imagine how that person may or may not look. There are also some Addons for Automatic1111 that may achieve what you want. You also have the option of using a Lora of a character that looks similar to what you want and change the look via prompting.

I recently dabbled into ComfyUI and this video popped up a while ago. It seems really close to what you search for.


ComfyUI has a node system. It has a steep learning curve but the control you have over your images is crazy. If I get more compfortable using ComfyUI then I will make maybe another tutorial.


Video AI models have the same issue as LLMs have or had in the past. They can link thoughts together for a few seconds until they loose the plot. For example image to video generation has come a long way but for local generation 5-10 seconds is quite the max.

A few days ago WAN2.1 has been released. You can use the WAN2.1 model in ComfyUI and make really great short AI videos from a single image and a few lines of prompts or only from text.

Oh nice, ComfyUI looks a lot like the sort of node scripting workflows that I'm actually used to from stuff like Unreal blueprints and material setups. I think I might find this more intuitive than using the SD webui.

I'll play around with it and see if I can find a good node setup that keeps the characters consistent. It seems to be okay if I use well-known existing characters (like Lara Croft or something) but it would be nice to be able to create my own cast.

I did a quick test last night of using Live2D to animate a sex scene, and it definitely seems possible. My attempt was quick and scrappy so it had seams and blurred edges, but I did technically manage to get a working looping sex gif in about an hour, from a SD generated prompt

 

[Psan2022]

So I started out using automatic1111 in January, using this thread as one of my guides. It all worked, but an image took ~20min to render - more than 30min if I used addetailer. I thought this was just cause my PC was bad, but I was happy that the results ended up looking good. A few days ago somebody mentioned that comfyui was a lot better when it came to speed, although a lot more complicated to get into. Images in comfyUI are created in 40-120 seconds, depending on whether I use a face detailer or the amount of Lora's.

I had expected some increase in speed, but I had not expected it running at 10 times the speed. This increase is so extreme that I am pretty certain I did something wrong when setting up automatic. Any ideas how to fix the speed issue? I can of course keep using comfyui, but automatic has some inbuilt features that where pretty neat, and it was a lot easier to use, so if I can get it to work better I would consider switching back.

Hmmm, thats really odd. I am also using ComfyUi for a short while now and I have not seen any drastic improvements in speed. I may be a little bit faster but not by much. It may be a rookie question but are you sure automatic1111 is running on your GPU and not CPU? Have you tried using --xformers in the webui.bat as a command?
The System Info Addon is quite neat:

You must be registered to see the links

When active you have a lot of information about your pc and what automatic1111 is using. On the right side you have this info:

You see the xformer version and if you are using your GPU to render -> active:cuda. If not then there will be someting regarding your CPU there.

 

[its_not_real]

Good thinking. I usually backup my venv when i have a working install (sometimes installing extentions can break the venv), but didnt know you could bring it over from A111 to Forge.

A bit old thread but... You guys realize that the venv directory is just the python module for virtual environments right?
If on windows, it's managed by conda and on linux, well `python -m venv venv` creates symlinks to whatever python environment you have set up to use (IIRC a1111 defaults and uses python3.10 if you let the .sh file do it for you at startup)
Backing up that dierctory makes little sense, you update it with pip (or rather, the script file does it for you when you start a1111. I havent used it in quite some time, I use forge, and I manage my pyenv manually since I play around with quite a few AI:s locally).

What you DO want to backup is addons, like if you for example downloaded and used ReActor before it was taken down by ghub, you might want to make sure THAT is not removed in an update.
Low chance risk if you update with `git pull`, but the chance risk increases to 100% if you use a release file...

 

[leerlauf]

Hmmm, thats really odd. I am also using ComfyUi for a short while now and I have not seen any drastic improvements in speed. I may be a little bit faster but not by much. It may be a rookie question but are you sure automatic1111 is running on your GPU and not CPU? Have you tried using --xformers in the webui.bat as a command?
The System Info Addon is quite neat:

You must be registered to see the links

When active you have a lot of information about your pc and what automatic1111 is using. On the right side you have this info:
View attachment 4604910
You see the xformer version and if you are using your GPU to render -> active:cuda. If not then there will be someting regarding your CPU there.

I guess there was something wrong with xformers. After reinstalling I am now at something between 2-7 minutes (speed seems to wildly differ even without any big changes to settings and prompts). Still considerably slower than comfyUI, but better than 20-30 minutes.

 

[its_not_real]

I guess there was something wrong with xformers. After reinstalling I am now at something between 2-7 minutes (speed seems to wildly differ even without any big changes to settings and prompts). Still considerably slower than comfyUI, but better than 20-30 minutes.

If you prefer the interface in a1111 over comfy, you can use forge that is the project that comfy copied code from to get it to generate faster.

I am not a big fan of people blindly recommending comfy UNLESS they understand and reads every python line in the nodes and workflows people download and use. Malicious code will execute without pretty much any protection (other than the USER knowing what they are doing).
It's a massive security risk waiting to happen. I have seen multiple "proof of concept" examples of ransomware being used in a node on comfy.

As long as you know what you are doing, fine. But blindly throwing people unaware of what I am even saying here into that project is border line malicious in itself IMHO...
Comfy is a very strong tool to use with automation for example, but for the single user working on one image at time, forge wins every day of the week.
Only downside is that a few comfy devs felt it necessary to harass the dev of forge (and controlnet for that matter) so he left the project. So it is hanging a little bit up in the air now, witch is VERY bad for the open source AI environment as a whole.
That dev has provided more to the open source AI space than the next 5 people combined...

I have both on my computer but still mostly use forge though, works awesome, even with flux. And I don't need to download or write fifty-eleven different nodes and workflows to use it.

 

[leerlauf]

If you prefer the interface in a1111 over comfy, you can use forge that is the project that comfy copied code from to get it to generate faster.

I am not a big fan of people blindly recommending comfy UNLESS they understand and reads every python line in the nodes and workflows people download and use. Malicious code will execute without pretty much any protection (other than the USER knowing what they are doing).
It's a massive security risk waiting to happen. I have seen multiple "proof of concept" examples of ransomware being used in a node on comfy.

As long as you know what you are doing, fine. But blindly throwing people unaware of what I am even saying here into that project is border line malicious in itself IMHO...
Comfy is a very strong tool to use with automation for example, but for the single user working on one image at time, forge wins every day of the week.
Only downside is that a few comfy devs felt it necessary to harass the dev of forge (and controlnet for that matter) so he left the project. So it is hanging a little bit up in the air now, witch is VERY bad for the open source AI environment as a whole.
That dev has provided more to the open source AI space than the next 5 people combined...

I have both on my computer but still mostly use forge though, works awesome, even with flux. And I don't need to download or write fifty-eleven different nodes and workflows to use it.

Awesome - forge was exactly what I was looking for. Works just as fast (or maybe even faster than) comfyui, without all the hassle with those nodes. The only thing I really miss about comfy is the ability to just queue up new image generations, but there is probably some extension for that somewhere.

 

[Psan2022]

I am not a big fan of people blindly recommending comfy UNLESS they understand and reads every python line in the nodes and workflows people download and use. Malicious code will execute without pretty much any protection (other than the USER knowing what they are doing).
It's a massive security risk waiting to happen. I have seen multiple "proof of concept" examples of ransomware being used in a node on comfy.

As long as you know what you are doing, fine. But blindly throwing people unaware of what I am even saying here into that project is border line malicious in itself IMHO...
Comfy is a very strong tool to use with automation for example, but for the single user working on one image at time, forge wins every day of the week.

Welp, did not realize Comfy can be that high of a risk, good to know

 

[its_not_real]

Welp, did not realize Comfy can be that high of a risk, good to know

Look, it's not a "risk" more than any other FOSS software you might use. As long as you don't pull workflows or nodes from "some random dude" but rely on big sites like hugginface and ONLY use stuff lots of others have downloaded, you're fine.
And with users that are not so knowledgeable, it's easy to follow some random dudes advice on reddit to "just download this workflow to get that to work"... You only need to be compromised once...

I did not want to discourage anyone from learning the software, as I said, its REALLY powerful if you know how to use it.
What I was trying to communicate is that for the end user, IMHO, it better to use a1111 or any of the forks out there.

Awesome - forge was exactly what I was looking for. Works just as fast (or maybe even faster than) comfyui, without all the hassle with those nodes. The only thing I really miss about comfy is the ability to just queue up new image generations, but there is probably some extension for that somewhere.

Most of the plugins exists in forks modified to work with forge.
You can find a whole bunch of them in

You must be registered to see the links

on ghub.

Edit
Wait a second, what do you mean by "only thing I really miss about comfy is the ability to just queue up new image generations". You can right click the "generate" button and choose "generate forever" and stop it with the button whenever you want.
Or there is x/y/z plot in the script dropdown at the bottom...

 

[leerlauf]

Look, it's not a "risk" more than any other FOSS software you might use. As long as you don't pull workflows or nodes from "some random dude" but rely on big sites like hugginface and ONLY use stuff lots of others have downloaded, you're fine.
And with users that are not so knowledgeable, it's easy to follow some random dudes advice on reddit to "just download this workflow to get that to work"... You only need to be compromised once...

I did not want to discourage anyone from learning the software, as I said, its REALLY powerful if you know how to use it.
What I was trying to communicate is that for the end user, IMHO, it better to use a1111 or any of the forks out there.



Most of the plugins exists in forks modified to work with forge.
You can find a whole bunch of them in

You must be registered to see the links

on ghub.

Edit
Wait a second, what do you mean by "only thing I really miss about comfy is the ability to just queue up new image generations". You can right click the "generate" button and choose "generate forever" and stop it with the button whenever you want.
Or there is x/y/z plot in the script dropdown at the bottom...

I already know about the x/y/z plot and generate forever, but I oftentimes want to make little adjustments when I am in the middle of a batch without having to interrupt generation completely. Xyz needs me to plan everything ahead, and generate forever just generates the given prompt forever (unless I misunderstood sth about it). With Comfy I could just switch out some prompts or loras on the go if a new idea popped into my head and add it all to the queue. Is there something like that for automatic/forge? The only thing I managed to find is the agent scheduler, which doesn't seem to work atm.

 

[Psan2022]

Look, it's not a "risk" more than any other FOSS software you might use. As long as you don't pull workflows or nodes from "some random dude" but rely on big sites like hugginface and ONLY use stuff lots of others have downloaded, you're fine.
And with users that are not so knowledgeable, it's easy to follow some random dudes advice on reddit to "just download this workflow to get that to work"... You only need to be compromised once...

I did not want to discourage anyone from learning the software, as I said, its REALLY powerful if you know how to use it.
What I was trying to communicate is that for the end user, IMHO, it better to use a1111 or any of the forks out there.

I am not discouraged by no means, only a smudge more careful thanks to you. Which is a good thing.
I am only using Comfy to generate videos with WAN2.1 right now, since Comfy has WAN2.1 integration. It takes a while but damn the results are promising

 

[its_not_real]

I already know about the x/y/z plot and generate forever, but I oftentimes want to make little adjustments when I am in the middle of a batch without having to interrupt generation completely. Xyz needs me to plan everything ahead, and generate forever just generates the given prompt forever (unless I misunderstood sth about it). With Comfy I could just switch out some prompts or loras on the go if a new idea popped into my head and add it all to the queue. Is there something like that for automatic/forge? The only thing I managed to find is the agent scheduler, which doesn't seem to work atm.

Not that I know of, but this is the exact kind of workflow that comfy excells at.
Your workflow is to generate a crapton of images, mine is usually to generate many txt2img:s in the beginning, then work more with inpainting, controlnet etc to refine one image to whatever I want.
ctrl+enter starts generation on a1111/forge, no need to scroll to the top to start generating.

But to me it sounds like comfy is better for you tbh.
Remember, there is also nothing stopping you from doing some things in comfy, and then use f.ex forge to refine. You use the same models and the png Meta info work the same for both.
Throw the generated image into "png info" and then press "use in txt2img|img2img" or wherever you want it and forge/a1111 will use the meta info to configure settings AND the image (remember to change seed to random unless you want the same img over and over after doing so).

 

[leerlauf]

Not that I know of, but this is the exact kind of workflow that comfy excells at.
Your workflow is to generate a crapton of images, mine is usually to generate many txt2img:s in the beginning, then work more with inpainting, controlnet etc to refine one image to whatever I want.

But to me it sounds like comfy is better for you tbh.
Remember, there is also nothing stopping you from doing some things in comfy, and then use f.ex forge to refine. You use the same models and the png Meta info work the same for both.

Yeah - that's what I wanted to do - use comfy to test out loras and styles and prompts, then switch to forge once I settled on something that generally works to create the final image. Was just worried when I heard that comfy is such a big security risk, but I assume I'll be fine as long as I only download and add the most well-known nodes. Thanks for all the help though. Forge feels like great upgrade compared to automatic.

 

[its_not_real]

Was just worried when I heard that comfy is such a big security risk

Again, not a "big security risk", especially not now when you think about it.
YOU are the security risk, comfy ui is working as intended.
If I were to make a risk analysis on you you went from "risk zone" to "99.9% safe" just by being informed to "be careful".
Just the same way you might tell your grandmother to "do not click links in emails" makes her significantly more safe from online malware. xD

I REALLY want to emphasize what I mean. Nodes run pretty much unchecked python code on your computer.
The biggest risk factor on any computer system is the user.
If the user is unfamiliar with the software that is being used, there is a WAY bigger risk of making dangerous and bad decisions, for example becoming frustrated on something not working as intended > search the internet > find a reddit thread/random forum > download something that then executes > bye bye.
And this goes for forge or any other software you run thb, it's just that it becomes SO much easier to for a malicious actor to get someone compromised because of the constant downloading of nodes in comfy...

Just be careful is what I am saying.

 

[a82912]

Not sure if there's a better place but figured I'd ask here:

Anyone know if there are public/FOSS libraries floating around for decensoring mp4s/moving images locally?

I've seen DeepMosaic, but the results don't look mind-blowing. Other sites seem to have achieved better results (rip hentai.ai or whatever it was called), and I imagine extending StableDiffusion or the like for the purpose would also be more viable in the long run.
PyInpaint does an admirable job considering the underlying concept but I doubt it will recover details well.

I'm especially interested if any of them are pytorch based and have the underlying structure in the clear, this is a hobby project but it never hurts to learn from what's out there.

 

[its_not_real]

Anyone know if there are public/FOSS libraries floating around for decensoring mp4s/moving images locally?

For automation like that, the thing me and the other dude just discussed, comfy ui. I would even go as far as to say it is the best for video out there right now. If it can do that stuff though? No idea, I have doubled very little with video generation, but in my head it should work (ie means nothing). xD
I don't think there is any "take_this_video_and_decensor_it" drop and play. You would have to make sure the models work as you intend tweak and fiddle.
At least that's what I think.

 

[Luncher]

A bit old thread but... You guys realize that the venv directory is just the python module for virtual environments right?
If on windows, it's managed by conda and on linux, well `python -m venv venv` creates symlinks to whatever python environment you have set up to use (IIRC a1111 defaults and uses python3.10 if you let the .sh file do it for you at startup)
Backing up that dierctory makes little sense, you update it with pip (or rather, the script file does it for you when you start a1111. I havent used it in quite some time, I use forge, and I manage my pyenv manually since I play around with quite a few AI:s locally).

What you DO want to backup is addons, like if you for example downloaded and used ReActor before it was taken down by ghub, you might want to make sure THAT is not removed in an update.
Low chance risk if you update with `git pull`, but the chance risk increases to 100% if you use a release file...

aha damn.. I'm pretty sure i just deleted reactor.. it never worked for me. though tbf i cant even set up koyha or onetrainer without running into issues.

About the venv though. Perhaps there is better ways, but I found that when messing around with extensions, sometimes you can run into conflicts and "break" the venv and the quickest way I've found to fix it is just to backup the venv, rather than creating a new one.

 

[its_not_real]

aha damn.. I'm pretty sure i just deleted reactor.. it never worked for me. though tbf i cant even set up koyha or onetrainer without running into issues.

Strange, still works fine for me, maybe you tried to use the a1111 version instead of the forge version..

About the venv though. Perhaps there is better ways, but I found that when messing around with extensions, sometimes you can run into conflicts and "break" the venv and the quickest way I've found to fix it is just to backup the venv, rather than creating a new one.

Whatever works for you is the correct way for you.
Conflicts can happen if you install things with pip manually instead of using the requirements file that is provided from github, this file is edited by the devs to make sure the correct versions of things are installed, and that stuff gets installed into the venv directory.
When starting the webui, the script checks for updates and makes sure dependencies are up to date, download and updates them if necessary.
Also: MAKE SURE THE VENV IS ACTIVATED IF YOU INSTALL THINGS WITH PIP or they will be installed on your machine, not inside the virtual environment!!! Not sure how you do that on window, on linux you just run the activate script inside ./venv/bin. I see there is a .PS script inside there, probably used on windows, but MAKE SURE YOU KNOW WHAT YOU ARE DOING!
What the venv does as I explained earlier, is to create a virtual environment inside that directory and point to a specific python version (using conda on windows). If it is activated and you run commands, it will look inside ./venv/bin first, for example pip or python and use that instead of your native versions of the same applications.
Take a look inside the venv/bin directory and you'll see (python is a link to python somewhere else). And dependencies downloaded is inside the venv/lib directory.
There might be a file called pyenv.cfg inside venv, if you open that file in a text editor it should tell you python version and where that python version is installed and linked to.

If a plugin is forcing you to break dependencies that the main project is using, you have to be prepared major things might break.
I generally discourage doing stuff like that unless you really know what you are doing.
Better to use plugins that are created to work with forge directly, not a1111 stuff that tends to lack behind with dependencies or "newer" things that require versions newer than what forge has in the requirements files.

Turned into a bit of a rant and a lesson, and things can differ a bit on windows, I don't have a win installation to check against.
But I hope someone finds it usefull.

 

[Luncher]

Strange, still works fine for me, maybe you tried to use the a1111 version instead of the forge version..



Whatever works for you is the correct way for you.
Conflicts can happen if you install things with pip manually instead of using the requirements file that is provided from github, this file is edited by the devs to make sure the correct versions of things are installed, and that stuff gets installed into the venv directory.
When starting the webui, the script checks for updates and makes sure dependencies are up to date, download and updates them if necessary.
Also: MAKE SURE THE VENV IS ACTIVATED IF YOU INSTALL THINGS WITH PIP or they will be installed on your machine, not inside the virtual environment!!! Not sure how you do that on window, on linux you just run the activate script inside ./venv/bin. I see there is a .PS script inside there, probably used on windows, but MAKE SURE YOU KNOW WHAT YOU ARE DOING!
What the venv does as I explained earlier, is to create a virtual environment inside that directory and point to a specific python version (using conda on windows). If it is activated and you run commands, it will look inside ./venv/bin first, for example pip or python and use that instead of your native versions of the same applications.
Take a look inside the venv/bin directory and you'll see (python is a link to python somewhere else). And dependencies downloaded is inside the venv/lib directory.
There might be a file called pyenv.cfg inside venv, if you open that file in a text editor it should tell you python version and where that python version is installed and linked to.

If a plugin is forcing you to break dependencies that the main project is using, you have to be prepared major things might break.
I generally discourage doing stuff like that unless you really know what you are doing.
Better to use plugins that are created to work with forge directly, not a1111 stuff that tends to lack behind with dependencies or "newer" things that require versions newer than what forge has in the requirements files.

Turned into a bit of a rant and a lesson, and things can differ a bit on windows, I don't have a win installation to check against.
But I hope someone finds it usefull.

Hey thanks man, you certainly taught learned a few things. Everything you said here does apply to windows as well. And its probably best to avoid installing into the venv manually unless you need to anyways, wouldnt you say? I try to avoid it.

Also, I haven't had any issues with forge itself TBF. I was thinking back a few years ago using the OG A1111. A big root of the issues I face with extensions seems to be due to AMD GPU support. its not as bad now, but reactor has never worked for me nor has koyha. And thats across A1111, vladmatic, forge, comfy, etc.

A bit of a random question, but do you know if it's worth it to quantize ones model collection? It would be nice if i could save some space or generate larger images, but not exactly worth it if theres a noticable difference in quality.

 

[its_not_real]

A bit of a random question, but do you know if it's worth it to quantize ones model collection? It would be nice if i could save some space or generate larger images, but not exactly worth it if theres a noticable difference in quality.

You mean like quantizise the models you downloaded?
I would not do that for a few reasons.
1. I use civitai, and civitai helper for example will no longer be able to link created images with models automatically (well, it actually would not be the same model any more)
2. If I want a lower Q model, I just download a lower Q model
3. It's probably cheaper to buy a new HD to fit the models on than spend the electricity to re-quantasize all your models (assuming you have more than a few models)

I do not have much knowledge about this so take my opinions with a grain of salt.

And as for GPU and AI generation, well.
You can say whatever you want about nvidia, but cuda cores makes nvidia king in this regard.

 

[idontjudgebro]

thanks for the tutorial it was awesome, helped a lot!

just started thinkering myself and had a couple of questions

is it better to have less prompts or the more detailled promtp possible ?

what are the best settings to done down to speed up image generation and mitigating the downgrade of the generated pictures? i want to dablle till i find the characters i 'like.

what's the process used from authors when generating IA to have consistantly the same characters face ?
for exemple when i find a characters i like through the IA generations, how can i make variants of it efficiently to use in a story?

If it's already answered just tell me where i can search thanks !