[Stable Diffusion] Prompt Sharing and Learning Thread

[rogue_69]

The Daz Gen 8 is a specific model that works with Face Transfer in Daz Studio. For this method to work, you would need to use some kind of 3D modeling software (blender, Daz, etc.). These allow you to pose, dress, change the hair, change expressions, and use the renders you get from these 3D models with IMG2IMG to get consistent characters. If you're interested, check out some YouTube videos on 3D modeled characters and see if it's something you're interested in learning how to do. You can get started with Daz Studio for free, and be able to use it to get some decent results with my workflow.

(Edit: Think of this workflow as using Stable Diffusion to re-texture Daz Studio images, rather than creating new images).

 

[Mr-Fox]

Not sure what you mean by "adding to".
IE

1. you train a lora on 20 images, then later you select that lora as starting point and continue the training with
   1. 20 new images OR
   2. the 20 old images along with 20 new ones (totaling 40 images)

1.2 would probably better off just retraining from scratch since doubling the amount of images would potentially change your learning quite a bit.

With 1.1 i suspect it would depend on how images were captioned, but i think it would "confuse" the AI. Generally the training picks up little to noting of the background unless there's "room left" to do so and "there's nothing left to learn" with the subject. Your images should have as different backgrounds as possible in your training set anyway which makes it less likely for the AI to associate background elements with the subject you're training.

As for multiple subjects in one lora, that should work just fine, assuming you can keep it all within the "size limits" of the lora. You have loras with subject wearing different clothing or styles, i can't remember seeing any with completely different people, but the training setup for it would be the same. The different image sets would just be in different folders and tagged accordingly.
(offtopic: One question is how "efficient" weights are being stored in the lora. IE if you have one character with and without a hat, would it then store just the differences between them or would it be stored as 2 complete "weight sets")
You'd probably need to make sure your sets are rather close in how fast the AI picks them up though, you can probably fix some of it with having different repeat counts for each set up just because repeats * images is the same it doesn't mean it'll learn at the same.
Through testing it seems that even running the same set at something like 4 repeats for 4 epochs doesn't give the same end result as doing 2 reps for 8 epochs. This could be due to "something" in the code itself that make it behave slightly different, i'd rather not dig into all that. If you're lucky enough to do batches as well, that throws in another curve ball apparently, anyway....

I'd planed to put multiple subjects into one lora but not gotten around to it yet as i were struggling to find a common setting so they'd all learn at roughly the same rate. I considered training one lora, on one set then "append" a new set to it, i started on it with the lora i posted a while back, but an update broke things so i can't even launch the trainer and i've not gotten back to it yet.
I remember seeing something about lora merging at the time too but not sure how that works

I have experimented just a little with Lora merging but had no good results from it. Probably would need more testing to reach any conclusion. Maybe I did it wrong, I merged different epochs, in other words the same Lora but with different amounts of epochs. I did this because it was what I had on hand atm and I wanted to see if I could get the best out of them. When you merge Loras you choose the merging ratio, lets say 0,6 of a and ofc 0,4 will be for b. An interesting test would be to merge the same Lora but that has been trained on different checkpoints, and/or different learning rates or other settings that yields different results, trying to get the best of both. For training multiple concepts or subjects, as far as I've read, you need to make sure that it's a mild/soft training for all so not one will be dominant and take over. The guide on rentry talks about this. I guess an even slower training is needed, perhaps fewer steps per epochs. Something else that would be interesting to try is something called "distill", it means training a Lora from the difference between two models.
The guides section on this is a bit murky and hard to understand though. My take on it is that you create a checkpoint from the difference of two models and then train a Lora on it. It involves setting up a virtual env, something I'm interested in but have no idea how it works.

You must be registered to see the links

You must be registered to see the links

 

[Mr-Fox]

I was messing around with training Daz Loras for a while, but I came up with a better workflow. Use a Daz Gen 8 headshot in Stable with prompts, take an image you like from that and use it with Face Transfer in Daz (use Face Transfer Shapes if you have it). Either used the texture from the Face Tranfer or use another texture, you're mainly just using the new Daz model you created for nose, mouth, eyes, and face consistency in Stable. Render out a bunch of images in Daz and bring them over to Stable and make images using the same prompts you used before.
Here is a quick example I threw together because I was bored. If I put more work into it, I could have gotten even more consistent results.
View attachment 2910699
View attachment 2910713
View attachment 2910707 View attachment 2910710

I just use roop instead with images of the web. You can edit them and prepare them in photoshop or similar first. This is what I did with the large post of all the celebrities a little while back.
https://f95zone.to/threads/stable-diffusion-prompt-sharing-and-learning-thread.146036/post-11409142

 

[Mr-Fox]

The Daz Gen 8 is a specific model that works with Face Transfer in Daz Studio. For this method to work, you would need to use some kind of 3D modeling software (blender, Daz, etc.). These allow you to pose, dress, change the hair, change expressions, and use the renders you get from these 3D models with IMG2IMG to get consistent characters. If you're interested, check out some YouTube videos on 3D modeled characters and see if it's something you're interested in learning how to do. You can get started with Daz Studio for free, and be able to use it to get some decent results with my workflow.

(Edit: Think of this workflow as using Stable Diffusion to re-texture Daz Studio images, rather than creating new images).

The face transfer is not specific to one Daz generation, it's simply a tool or function in daz studio. The daz generations is the basemodels that the developers create as they update daz studio, this is then what everything else i based on, now they have Gen 9. You can use a front image and a side image (profile) and use something like

You must be registered to see the links

to create a 3d model of that face and import it to daz studio with face transfer. It creates a morph for this face and you also import the textures that

You must be registered to see the links

creates.

Facegen:


Example:

For this render I used this face:


Roop is a much simpler approach though and it does a very good job even in different poses.

 

[Mr-Fox]

Just a link for a post I made about roop and the code edit you need to turn of the nsfw filter.
https://f95zone.to/threads/stable-diffusion-prompt-sharing-and-learning-thread.146036/post-11443616

A few examples:

London Keys	Alicia Vikander	Jennifer Lawrence	Gabrielle Union

I used roop with img2img, also known as photobashing. The face is only copy pasted over the original with this method so the shape is not changed. However when you use roop with txt2img roop is included in the generative process so the face shape is changed. I guess with a higher denoising setting you could possibly get a better result with img2img, I only did this very fast as an example.

 

[Mr-Fox]

Another example of using roop but with txt2img instead of img2img. I used the same pose with openpose and the same promt and seed, so the likeness isn't as good as it could be. Again just a very quick example.

London Keys	Alicia Vikander	Jennifer Lawrence	Gabrielle Union

 

[Sepheyer]

I might change this later, this is my "first impressions".
---
Do HoneySelect2 LORAs make sense?

(WTF is HoneySelect2: DAZ for smoothbrains.)

Rather yes, but with caverats. I mean with caveats. So, I just trained LORA off an HS2 chara Sina. The SD renders using this LORA require time figuring out what works: the LORA was not immediately plug-n-play, I nearly wrote it off at first. It requires going through various checkpoints, samplers and resolutions to determine which ones work good with this LORA.

Here's what the HoneySelect2 model looks like, there were 58 images in total, took ~16 hours (although the original estimate was 28 hours). The images LORA was trained on are all 512x512 with WD14 automatic caption. At first I manually amended the captions removing some tags, adding others. Then I said "fuck it", and re-run overwriting the manual inputs by using completely automatic default captioning as I wanted to see what an entirely automatic captioning would look like.



(Mr-Fox you are no longer collecting royalties for that white-and-yellow workout outfit, right? Good.)

The SD 512x512 renders of this LORA are straight up trash, but for unrelated reasons. The issue is the face restore module where the somewhat covered face makes face restore glitch on such a small resolution.

Spoiler: 512x512 Renders

You don't have permission to view the spoiler content. Log in or register now.

But the renders improve as the resolution doubles:

Spoiler: More

You don't have permission to view the spoiler content. Log in or register now.

PS I might be wrong about the following conclusion. The LORA SophiaCollegeFace (discussed in posts above) was trained on a variety of render sizes: 512x512, 768x512, etc. And the LORA didn't render really well in higher resolution - at least out of five renders none were usable, although the face and breast came out great:

Spoiler: Illustration

You don't have permission to view the spoiler content. Log in or register now.

But this HS2 LORA trained exclusively on 512x512 renders has a really high success ratio specifically for higher resolution renders, although has completely crappy 512x512 renders. There pretty much I got a 10/10 hit ratio.

And finally, there is an issue of background contamination, where the checkpoint's background gets overwritten by the LORA's background. See for yourself, the greenery are the low-quality legacy of the HoneySelect2 renders:

Spoiler: Illustration

You don't have permission to view the spoiler content. Log in or register now.

Let me know if you have any ideas how to fix this and restore the checkpoint's original backgrounds. The one solution I can think of is to avoid using such images in the training dataset, and instead ensure that the entire dataset has rather rare or abstract backgrounds.

 

[Mr-Fox]

I might change this later, this is my "first impressions".
---
Do HoneySelect2 LORAs make sense?

(WTF is HoneySelect2: DAZ for smoothbrains.)

Rather yes, but with caverats. I mean with caveats. So, I just trained LORA off an HS2 chara Sina. The SD renders using this LORA require time figuring out what works: the LORA was not immediately plug-n-play, I nearly wrote it off at first. It requires going through various checkpoints, samplers and resolutions to determine which ones work good with this LORA.

Here's what the HoneySelect2 model looks like, there were 58 images in total, took ~16 hours (although the original estimate was 28 hours). The images LORA was trained on are all 512x512 with WD14 automatic caption. At first I manually amended the captions removing some tags, adding others. Then I said "fuck it", and re-run overwriting the manual inputs by using completely automatic default captioning as I wanted to see what an entirely automatic captioning would look like.

View attachment 2910747 View attachment 2910748 View attachment 2910749

(Mr-Fox you are no longer collecting royalties for that white-and-yellow workout outfit, right? Good.)

The SD 512x512 renders of this LORA are straight up trash, but for unrelated reasons. The issue is the face restore module where the somewhat covered face makes face restore glitch on such a small resolution.

Spoiler: 512x512 Renders

You don't have permission to view the spoiler content. Log in or register now.

But the renders improve as the resolution doubles:

View attachment 2910801

Spoiler: More

You don't have permission to view the spoiler content. Log in or register now.

PS I might be wrong about the following conclusion. The LORA SophiaCollegeFace (discussed in posts above) was trained on a variety of render sizes: 512x512, 768x512, etc. And the LORA didn't render really well in higher resolution - at least out of five renders none were usable, although the face and breast came out great:

Spoiler: Illustration

You don't have permission to view the spoiler content. Log in or register now.

But this HS2 LORA trained exclusively on 512x512 renders has a really high success ratio specifically for higher resolution renders, although has completely crappy 512x512 renders. There pretty much I got a 10/10 hit ratio.

And finally, there is an issue of background contamination, where the checkpoint's background gets overwritten by the LORA's background. See for yourself, the greenery are the low-quality legacy of the HoneySelect2 renders:

Spoiler: Illustration

You don't have permission to view the spoiler content. Log in or register now.

Let me know if you have any ideas how to fix this and restore the checkpoint's original backgrounds. The one solution I can think of is to avoid using such images in the training dataset, and instead ensure that the entire dataset has rather rare or abstract backgrounds.

(Mr-Fox you are no longer collecting royalties for that white-and-yellow workout outfit, right? Good.)

Royalties? Never got jack doo doo for anything I have created, other than likes and possibly good will.

Don't forget to try out your Lora at different weights. Keep the prompt very simple. Try to exclude as many variables as possible. When training it's a good idea to select the setting for saving a Lora for each epoch, this way you can set it to lets say 5 epochs and can then do tests with xyz plot and choose the one that is giving you the best consistent results. The settings in schlongborns guide post is based on the generic YT guides by the likes of Sebastian Kamph and Aitrepeneur. It's fine for getting you going as a trial run but it's never going to give very good results. Instead go to the rentry thread I linked to and copy his settings and use as your base, then you need to adjust the settings according to your needs. It took me many trial runs before I got decent settings and consistent results. Don't shy away from a very long training time, it's better to have a somewhat mild or weak Lora then a completely overcooked one that you can't do shite with.

Here's my preset for my training. You will need to change the folder locations etc and also the settings for your own purposes but it's something to start with.

 

[me3]

I might change this later, this is my "first impressions".
---
Do HoneySelect2 LORAs make sense?

...

And finally, there is an issue of background contamination, where the checkpoint's background gets overwritten by the LORA's background. See for yourself, the greenery are the low-quality legacy of the HoneySelect2 renders:

Spoiler: Illustration

You don't have permission to view the spoiler content. Log in or register now.

Let me know if you have any ideas how to fix this and restore the checkpoint's original backgrounds. The one solution I can think of is to avoid using such images in the training dataset, and instead ensure that the entire dataset has rather rare or abstract backgrounds.

if the background (or other elements) you don't want is bleeding into the lora it's generally a sign of needing to be "captioned out" in the training and/or you've "overtrained" in the sense that the AI has already learned the subject and it's including other things just because they are common, not captioned out and there's "room left to fill" in the lora. Simple thing might just be to say the images are in a forest or park

 

[Mr-Fox]

Penny wants to try out the new bed..

 

[Sepheyer]

Imma start on Lara Croft LORA: https://f95zone.to/threads/loras-for-wildeers-lara-croft-development-thread.173873/

 

[me3]

Imma start on Lara Croft LORA: https://f95zone.to/threads/loras-for-wildeers-lara-croft-development-thread.173873/

RE this, i had to get the training stuff working again so i grabbed some of the images for a quick test (quick as in prep, not actual AI training).
Seeing some issues with the captions and potentially my choice of images used, but it seems to be working. One (or more) images clearly has some wet/oily skin that's not captioned out, but hey maybe it's sweaty in the jungle, not that uncommon


(image got resized despite me cutting it up, i fully understand the byte limit. but less of a fan of the pixel limit, anyway...It's only by a few hundred px)

 

[Sharinel]

RE this, i had to get the training stuff working again so i grabbed some of the images for a quick test (quick as in prep, not actual AI training).
Seeing some issues with the captions and potentially my choice of images used, but it seems to be working. One (or more) images clearly has some wet/oily skin that's not captioned out, but hey maybe it's sweaty in the jungle, not that uncommon
View attachment 2915252

(image got resized despite me cutting it up, i fully understand the byte limit. but less of a fan of the pixel limit, anyway...It's only by a few hundred px)

Is this Wildeer's Lara? Must say the face doesn't look like it in any of the pics to me?

 

[me3]

Is this Wildeer's Lara? Must say the face doesn't look like it in any of the pics to me?

can't say, all images are from that thread but i have no idea who made them. images aren't consistent in features and some of them have watermarks suggesting it's different creators. If you go though all the images you clearly see there's differences both in body and face.

 

[Sepheyer]

can't say, all images are from that thread but i have no idea who made them. images aren't consistent in features and some of them have watermarks suggesting it's different creators. If you go though all the images you clearly see there's differences both in body and face.

That's actually very-very good.

 

[Sepheyer]

RE this, i had to get the training stuff working again so i grabbed some of the images for a quick test (quick as in prep, not actual AI training).
Seeing some issues with the captions and potentially my choice of images used, but it seems to be working. One (or more) images clearly has some wet/oily skin that's not captioned out, but hey maybe it's sweaty in the jungle, not that uncommon
View attachment 2915252

(image got resized despite me cutting it up, i fully understand the byte limit. but less of a fan of the pixel limit, anyway...It's only by a few hundred px)

Can you please post in the Lara's LORA thread:
- which images you used (just a screenshot of your training image folder)
- what your tags look like
- what model was used for training (ideally if you could post your Kohya settings)
- one of the files that contains a prompt
- and how you got the green leotard to come up

 

[me3]

Can you please post in the Lara's LORA thread:
- which images you used (just a screenshot of your training image folder)
- what your tags look like
- what model was used for training (ideally if you could post your Kohya settings)
- one of the files that contains a prompt
- and how you got the green leotard to come up

Don't have time right atm, but will try later. As quick run down:
- <tbd>
- horrible, it was just blip without beams as that is still broken for me, yay
- settings is exactly the same as i used in the lora i posted a while ago, constant, 0.0001 lr, 128/64 network, adamw 8bit, cyber_3.3, can't remember there being much else settings atm
- prompt for generation was basically just "woman dressed for jungle exploration", AI clearly think you don't need too much clothing, probably due to heat and you see she looks drenched in sweat
- "nothing", coloring the AI picked up itself, it really likes using it for clothing, rest is more then likely to not having captioned anything about the clothing. captions are so basic they mentioned next to nothing besides, woman, <optional pose>, <optional location>. Lack of beams with blip is horrible.

 

[Sepheyer]

Don't have time right atm, but will try later. As quick run down:
- <tbd>
- horrible, it was just blip without beams as that is still broken for me, yay
- settings is exactly the same as i used in the lora i posted a while ago, constant, 0.0001 lr, 128/64 network, adamw 8bit, cyber_3.3, can't remember there being much else settings atm
- prompt for generation was basically just "woman dressed for jungle exploration", AI clearly think you don't need too much clothing, probably due to heat and you see she looks drenched in sweat
- "nothing", coloring the AI picked up itself, it really likes using it for clothing, rest is more then likely to not having captioned anything about the clothing. captions are so basic they mentioned next to nothing besides, woman, <optional pose>, <optional location>. Lack of beams with blip is horrible.

Great. My LORA is still training, around 20 hours left out of 50. Gonna suck major dick if I picked bad settings.

Meanwhile, I did a few renders without the LORA asking for Lara Croft / Tomb Raider. All of them are pretty far from Wildeer's, so we'll know right away is the LORA comes out aight.

The leotard is what makes me uneasy. I hope the LORA gets it out of the box, but I know I put zero effort there. Now debating if the second run should have tags such as: WildeerLara nude / WildeerLara in leotard and shorts / WildeerLara in leotard. Gonna be an excuse to learn the Kohya's "continue training" feature with a smaller subset on top of an already baked LORA.

Spoiler: More

You don't have permission to view the spoiler content. Log in or register now.

 

[rogue_69]

Just a link for a post I made about roop and the code edit you need to turn of the nsfw filter.
https://f95zone.to/threads/stable-diffusion-prompt-sharing-and-learning-thread.146036/post-11443616

A few examples:

[TR]
[TD]London Keys[/TD]
[TD]Alicia Vikander[/TD]
[TD]Jennifer Lawrence[/TD]
[TD]Gabrielle Union[/TD]
[/TR]
[TR]
[TD] View attachment 2910883 [/TD]
[TD] View attachment 2910884 [/TD]
[TD] View attachment 2910885 [/TD]
[TD] View attachment 2910886 [/TD]
[/TR]

I used roop with img2img, also known as photobashing. The face is only copy pasted over the original with this method so the shape is not changed. However when you use roop with txt2img roop is included in the generative process so the face shape is changed. I guess with a higher denoising setting you could possibly get a better result with img2img, I only did this very fast as an example.

I just got done messing around with Roop, but I'll have to check out turning off the NSFW filter. My end goal is doing animations, which I've gotten good results already, just not as good as I want. I think Roop might really help. I'm happy with the renders of my character's bodies, clothing, the background, etc. It's just touching up the face that I use Stable Diffusion for. I'm thinking Roop really might help eliminate more of the flicker you get with AI images and animation.

 

[Mr-Fox]

I just got done messing around with Roop, but I'll have to check out turning off the NSFW filter. My end goal is doing animations, which I've gotten good results already, just not as good as I want. I think Roop might really help. I'm happy with the renders of my character's bodies, clothing, the background, etc. It's just touching up the face that I use Stable Diffusion for. I'm thinking Roop really might help eliminate more of the flicker you get with AI images and animation.

It's very easy to edit the code to turn off the nsfw filter.
Go to: stable-diffusion-webui\extensions\sd-webui-roop\scripts\

cimage.py

either right mouse click and choose edit or open an empty notepad doc and drag/drop the file on to notepad.
Erase all code in cimage.py
Copy/paste this code instead (minus the first and last quotation marks""):

"import tempfile
def convert_to_sd(img):
return [False, tempfile.NamedTemporaryFile(delete=False, suffix=".png")]"

then save, done!

The nsfw filter is really janky and might give false positives as well as missing others, so it's not only to bypass the filter that we edit the code, it will make roop function much better in all scenarios.