[Stable Diffusion] Prompt Sharing and Learning Thread

[hkennereth]

You can also do this either with tag mixing or keyword weighting.

Tag mixing:

View attachment 2987332
As you can see it's essentially like using a "refiner". The first person is the main checkpoint and the second is the "refiner" but the weight is reversed. This means that if you want the "refiner" to have more impact you need to increase the number. If you want the checkpoint to have more power the opposite is true. Don't use "[ ]", use normal brackets "( )" instead. It tend to give you and error with SD if you use "[ ]" .

Keyword weighting:

This can be used for blending more than 2 faces.
(Emma Watson:0.5), (Tara Reid:0.9), (Ana de Armas:1.2)
View attachment 2987341

Source:

You must be registered to see the links

The tag mixing shown on your first image does work, however it is as far as I know a feature of Automatic1111, and it's either not supported or works differently on other UIs since it changes how the diffusion process works on that image. ComfyUI, my app of choice, doesn't really support that, and I didn't find that trying to change weights works as reliably as I'd like, the results are not as consistent across a wide range of images, which is the point of my original post.

The method I suggested is more flexible and works on any image generation app since it's just basic prompting. It doesn't allow the same level of control, of course, but it is in my experience better for getting a consistent "new" person across many images even when changing styles or checkpoints. Just something to keep in mind.

 

[Mr-Fox]

The tag mixing shown on your first image does work, however it is as far as I know a feature of Automatic1111, and it's either not supported or works differently on other UIs since it changes how the diffusion process works on that image. ComfyUI, my app of choice, doesn't really support that, and I didn't find that trying to change weights works as reliably as I'd like, the results are not as consistent across a wide range of images, which is the point of my original post.

The method I suggested is more flexible and works on any image generation app since it's just basic prompting. It doesn't allow the same level of control, of course, but it is in my experience better for getting a consistent "new" person across many images even when changing styles or checkpoints. Just something to keep in mind.

You get the most consistent result with something like roop or controlnet models since it's the same input every time. I don't know if these are available for comfyui though.

 

[hkennereth]

You get the most consistent result with something like roop or controlnet models since it's the same input every time. I don't know if these are available for comfyui though.

I don't think that ControlNet is really a good solution for this particular problem, as it allow following a pre-existing composition, but not create a new image only by prompt with the flexibility that this gives as far as completely different result each time you run the prompt. For example, those images of the girl on the bar on my original example were the result of the exact same prompt, I just asked it to generate X images, without needing to prepare any source images for it. That said, if you do want a specific composition, you can surely use ControlNet in addition to the technique above, and yes, ControlNet is available on ComfyUI.

The best alternative is really using a LoRA or Dreambooth so you can train a model to create images of that specific person, but that is better used when you want to reproduce a pre-existing person, not a new fictional one. So if you want to make a game with Angelina Jolie as your main character, training a LoRA or Dreambooth model of her would be the best solution for sure. But that does require a lot of work. My suggestion, or yours of using tag mixing, are better when you want to create a new character "from scratch", and you're just providing some "DNA" to help direct the prompt make that same character consistently.

I know nothing about "roop", so I can't really speak much to it as a good alternative here, but from a very quick look at their GitHub page it also seems better for cases where you're trying to reproduce an existing real person, not create one from scratch. Please correct me if I'm wrong.

 

[Mr-Fox]

I don't think that ControlNet is really a good solution for this particular problem, as it allow following a pre-existing composition, but not create a new image only by prompt with the flexibility that this gives as far as completely different result each time you run the prompt. For example, those images of the girl on the bar on my original example were the result of the exact same prompt, I just asked it to generate X images, without needing to prepare any source images for it. That said, if you do want a specific composition, you can surely use ControlNet in addition to the technique above, and yes, ControlNet is available on ComfyUI.

The best alternative is really using a LoRA or Dreambooth so you can train a model to create images of that specific person, but that is better used when you want to reproduce a pre-existing person, not a new fictional one. So if you want to make a game with Angelina Jolie as your main character, training a LoRA or Dreambooth model of her would be the best solution for sure. But that does require a lot of work. My suggestion, or yours of using tag mixing, are better when you want to create a new character "from scratch", and you're just providing some "DNA" to help direct the prompt make that same character consistently.

I know nothing about "roop", so I can't really speak much to it as a good alternative here, but from a very quick look at their GitHub page it also seems better for cases where you're trying to reproduce an existing real person, not create one from scratch. Please correct me if I'm wrong.

If you have not tried the tools or techniques I'm talking about then you can't know the potential. I would suggest that you try them before writing it off or assuming the best use case scenario. Just to be clear I'm not yelling at you, I'm only stating my thoughts and opinions. Roop is very useful for creating a character from scratch also, not only to make fakes of real people. I am suggesting that if you generate a face only, SD will have more resources to give it more detail and quality, then you can use roop with this generated face to generate the entire character with body. In this scenario you would get much nicer faces with nice looking bodies as well. Ofc you can use any tag mixing or other method that works for you when you generate the face portrait. The new controlnet has a model named ip adapter. With this you can use a character that you have created and then simply change the composition or pose, this makes it very consistent and you can make your dataset for a Lora or checkpoint this way. I think it is available for comfy also. Openpose has a face only model, with this you will only give SD the bare bones so to speak, SD will still generate different results with every seed. Or you can of course use the full openpose model and get the same but with the body also. These tools can be used in many ways. They will not limit you, it's only about how creative you can be in using the tools and how imaginative you are.

Here's a demo/tutorial video for Ip adapter:

You must be registered to see the links

 

[hkennereth]

If you have not tried the tools or techniques I'm talking about then you can't know the potential. I would suggest that you try them before writing it off or assuming the best use case scenario. Just to be clear I'm not yelling at you, I'm only stating my thoughts and opinions. Roop is very useful for creating a character from scratch also, not only to make fakes of real people. I am suggesting that if you generate a face only, SD will have more resources to give it more detail and quality, then you can use roop with this generated face to generate the entire character with body. In this scenario you would get much nicer faces with nice looking bodies as well. Ofc you can use any tag mixing or other method that works for you when you generate the face portrait. The new controlnet has a model named ip adapter. With this you can use a character that you have created and then simply change the composition or pose, this makes it very consistent and you can make your dataset for a Lora or checkpoint this way. I think it is available for comfy also. Openpose has a face only model, with this you will only give SD the bare bones so to speak, SD will still generate different results with every seed. Or you can of course use the full openpose model and get the same but with the body also. These tools can be used in many ways. They will not limit you, it's only about how creative you can be in using the tools and how imaginative you are.

Here's a demo/tutorial video for Ip adapter:

You must be registered to see the links

Of course. Out of those only Roop is the one I'm not familiar with, mostly because I don't really use A1111 anymore, and I don't think it's available for ComfyUI. But I am familiar with IP Adapter, and while I haven't been making a ton of art lately, I have had the chance to play around with it and found that it has a ton of potential. The images below were made with it, and a SD1.5 model.

 

[Fuchsschweif]

Do I have to pay for stable diffusion membership in order to do all of that or is downloading the github stuff alone enough already?

 

[me3]

It's free.
Not sure which UI/system you're planing to use but both

You must be registered to see the links

and

You must be registered to see the links

is pretty easy to get to run and use, a1111 being the simplest to use of the two
There are those that sell models and loras, but that really isn't worth even considering for 99% of the usage. You can get very good models on sites like

You must be registered to see the links

, same with loras. There's obviously badly made ones too but you usually notice those either by images, comments and amounts of downloads.

 

[Fuchsschweif]

It's free.
Not sure which UI/system you're planing to use but both comfyui and automatic1111 is pretty easy to get to run and use.
There are those that sell models and loras, but that really isn't worth even considering for 99% of the usage. You can get very good models on sites like

You must be registered to see the links

, same with loras. There's obviously badly made ones too but you usually notice those either by images, comments and amounts of downloads.

So basically it's free because I have to "train" it by myself first? Or are models some kind of pre-trained modules?

 

[Jimwalrus]

So basically it's free because I have to "train" it by myself first? Or are models some kind of pre-trained modules?

SD is completely free because Hugging Face released it as such!
I know, right?
The only possible expense* with using Civitai is some creators will set their models as 'Early Release' for which you have to be a paid member to access for a small number of days.
No training is required to use SD, including any of the models on Civitai or elsewhere (they are, as you put it "pre-trained modules"), but you're free to follow tutorials and do some training if you wish. It's also possible you'll want something that no-one else has trained yet.


*Electricity bills aside - they shouldn't be too much unless you have a crazy multi-GPU set up

 

[Fuchsschweif]

SD is completely free because Hugging Face released it as such!
I know, right?
The only possible expense* with using Civitai is some creators will set their models as 'Early Release' for which you have to be a paid member to access for a small number of days.
No training is required to use SD, including any of the models on Civitai or elsewhere (they are, as you put it "pre-trained modules"), but you're free to follow tutorials and do some training if you wish. It's also possible you'll want something that no-one else has trained yet.


*Electricity bills aside - they shouldn't be too much unless you have a crazy multi-GPU set up

I somehow thought SD is Dalle2 and since they charge premium that there is no free version. Or is Dalle just another big model that gives you external rendering, making use of SD, therefore charge you?

I've got a GTX 1070, does it make sense to generate pictures with that or will it take an eternity to get pictures generated?

Thanks for all the info

 

[me3]

I somehow thought SD is Dalle2 and since they charge premium that there is no free version. Or is Dalle just another big model that gives you external rendering, making use of SD, therefore charge you?

I've got a GTX 1070, does it make sense to generate pictures with that or will it take an eternity to get pictures generated?

Thanks for all the info

Considering i'm using a 1060 (and 1050 in some cases), you should to pretty fine on a 1070

 

[Fuchsschweif]

And the NSFW restrictions that I face with e.g. midjourney or dalle 2 aren't present when using SD locally?

Edit: What's the difference between something like "Stable Diffusion v1-5 Model Card" and the models that I find on Civitai?

Is the first one more of an allrounder model and the models on Civitai are specific things only?

 

[me3]

v1-5 is the base model so it's just a basic a bit of everything model, it can create very good images but models of civitai will generally be better as long as you choose one that's meant for what you want to create. IE pure anime models is unlikely to create good photo realistic images.

As for NSFW most models you find on that site probably creates it in some way, there are more specific ones for it too, often including some kind of reference to it in their name...IE porn...

 

[Fuchsschweif]

v1-5 is the base model so it's just a basic a bit of everything model, it can create very good images but models of civitai will generally be better as long as you choose one that's meant for what you want to create. IE pure anime models is unlikely to create good photo realistic images.

As for NSFW most models you find on that site probably creates it in some way, there are more specific ones for it too, often including some kind of reference to it in their name...IE porn...

But can I just pick any random (character) model that I like and generate NSFW pictures with it? I'm just wondering where SD gets the "info" from. So let's say I have only installed the base modell v1-5 and then download a model of famous anime character X (not porn specifically).

Now I want to use that model to put that character nude into scenario Y. Does that simply work, and if so, where does SD then gather the information on how to create that picture - from the base model? Or do I need to get a somewhat porn-database-model for SD to be able to pull that off?

*Electricity bills aside - they shouldn't be too much unless you have a crazy multi-GPU set up

Is there a way to limit the GPU usage to be on the safe side, or is a single GPU setup not worth tweaking in terms of expenses?

 

[me3]

The model/checkpoint is what contains most of the data/"knowledge". So if you want to make anything nude or more sexual oriented you need a model that has that knowledge. So in that case the base stable diffusion 1.5 isn't the best suited, it should work but not sure how well the end result would be.
When you're specifically talking about a "model of a anime character" you're probably thinking of something LoRA or an embedding. This are more like addons that provide specific instructions to the model. Sort of like a specialist coming in to guide the end result.

 

[Fuchsschweif]

The model/checkpoint is what contains most of the data/"knowledge". So if you want to make anything nude or more sexual oriented you need a model that has that knowledge. So in that case the base stable diffusion 1.5 isn't the best suited, it should work but not sure how well the end result would be.
When you're specifically talking about a "model of a anime character" you're probably thinking of something LoRA or an embedding. This are more like addons that provide specific instructions to the model. Sort of like a specialist coming in to guide the end result.

Okay but are models closed or can my stable diffusion just access all models I have installed?

E.g. I install a model specialized in nude stuff and then use the anime character X model for nude generations, does it then combine data?

 

[Fuchsschweif]

Ok something is terribly off

So I downloaded this model:

You must be registered to see the links

It's installed in the "Lora" folder under models. I picked it in my SD UI and just went with a simple prompt to see the output. This is the result:



This doesn't even look remotely like the model

Here are my settings. What did I do wrong?

 

[me3]

Ok something is terribly off

So I downloaded this model:

You must be registered to see the links

It's installed in the "Lora" folder under models. I picked it in my SD UI and just went with a simple prompt to see the output. This is the result:

View attachment 2992823

This doesn't even look remotely like the model

Here are my settings. What did I do wrong?

View attachment 2992824


Edit: Or did it use the basic SD model? I think it might, because I switched the downloaded model from the model folder to the LORA folder.
But even with the same setting before, when it was in the model folder, and I specifically picked the downloaded model, I got weird results that didn't even remotely looked like anime.

is the lora called just "ina"? to easily use loras you can go into the lora tab and click it there. Also loras very often have trigger words or combination of words. If you look at the right side on the loras page on civitai you see it lists multiple trigger combinations, you need to use one of those in your prompt.

As a general warning be VERY careful about the word "girl" in your prompt. It has a high chance of giving you a underage character, which would be very bad in the case of NSFW images

 

[Fuchsschweif]

is the lora called just "ina"? to easily use loras you can go into the lora tab and click it there. Also loras very often have trigger words or combination of words. If you look at the right side on the loras page on civitai you see it lists multiple trigger combinations, you need to use one of those in your prompt.

It's called "ninomae ina'nis 5 outfits.safetensors". Your reply did overlap with my edit, if you watch my post again now you can see my settings, I tried the exact suggested combination of words. That's the only file that's downloadable from that page.

But shouldn't it work with any input, isn't that the main idea behind the whole thing, that we can create whatever we want?

 

[me3]

It's called "ninomae ina'nis 5 outfits.safetensors". Your reply did overlap with my edit, if you watch my post again now you can see my settings, I tried the exact suggested combination of words. That's the only file that's downloadable from that page.

But shouldn't it work with any input, isn't that the main idea behind the whole thing, that we can create whatever we want?

The LoRA you downloaded seem to be to create one character with different "outfits", so it seems to be very specific in what it wants to do.
You might be able to trigger just the character if you remove the outfit parts from the prompt, but i doubt it'll work fully.
There might be other LoRAs for the same character which is less restricted