I posted in the AI art thread about training a LORA. I used this project:
You must be registered to see the links
which is like webui, but only for training. I found it easier to use then dreambooth (which also constantly broke my webui installation, so I uninstalled it). It is a dedicated thing just for training so you can have webui for the art, and kohya for the training.
Most difficult thing was putting together a training set. I just used a series of Daz renders I have:
You must be registered to see the links
(I am actually too lazy to render more so I just keep reusing that one for my AI shenanigans since its kind of complete).
And then I cropped the images into "AI friendly" resolutions, so 512x512, 768x512, 1024x512 etc. I used mostly 512x512 pictures which is recommended, but it should be ok to use other resolution, even completely different ones from these "AI friendly" ones.
Kohya expects a certain directory structure for training, like this:
View attachment 2470222
and then inside of "image" you need to create a directory like so:
View attachment 2470223
That folder name is important like that, the 100 means how often each picture is sampled. So in my case I have 21 pictures, and 100 samples (or iterations) per picture, so the entire training will run for 2100 iterations. You can put higher values there if you have less images, youtube said around 1000-2000 iterations is enough.
and in there you can put all the images like so:
View attachment 2470224
And then in kohya, you can caption those images using BLIP like so:
View attachment 2470231
The prefix there is optional, that string is just inserted at the beginning of each generated .txt file. This is useful because then the LORA is trained to recognize that word, so when I put "SophiaCollegeLectureFace" in my prompt, or negative prompt, then I can control my LORA that way. I am not actually sure this really worked, I just used the LORA through webui's "additional networks" which just added something to the prompt anyways. But it didn't hurt either.
Ok, once you ran BLIP the folder should look like this:
View attachment 2470245
I edited all of these files because for my training data it all came out pretty much similar. To be honest I think my training data was kinda shitty. But it is just an example anyways. I'll attach a zip with my training data and the LORA, so there you can take a look what I used, most of these .txt files contain something like: "SophiaLectureCollegeFace a woman with red hair with glasses and a necklace on her neck".
Ok, now that everything is setup, you can go into the "Dreambooth LORA" tab and setup the folders like so:
View attachment 2470226
and also pick a source model like so:
View attachment 2470249
I just picked a v1.5 model, if you want to train a LORA for v2.1 or v2.0, you need to tick the v2 checkbox, if you want to train for v2.1-768 you need to also tick v_parameterization. Since I use a custom checkpoint, the model quick pick is "custom", otherwise you can select the base 1.5, 2.0 or 2.1 models there. I just save as safetensors because thats the default.
And then finally you can fiddle with the training parameters:
View attachment 2470258
View attachment 2470264
this what I used for the basics, and the advanced stuff I pretty much left everything as default.
One important thing is probably the resolution, I don't think I messed with it (made these screenshots after the fact so don't remeber exactly), but I wouldn't change it too much either way. 512x512 or 768x768 is probably the way to go there with a 1.5 model, maybe 768x768 only really works with v2.1-768 even. Might also depend on your training data. If you have only 768x768 images then maybe thats the better choice.
The other important one is the "Enable buckets" checkbox. That makes it so that your training data is sorted into resolution "buckets", not entirely sure how that works but it is meant to make it so you can use images with differing resolutions, or resolutions that don't "fit" the model way in your training data without distorting the result. So I'd make sure it is enabled (I believe it costs VRAM though).
And now, you can press the big orange button!
View attachment 2470267
And your training should start.
Code:
(venv) → D:\AI\kohya_ss [master ≡ +1 ~0 -0 !]› .\webui-user.bat
Already up to date.
Validating that requirements are satisfied.
All requirements satisfied.
Load CSS...
Running on local URL: http://127.0.0.1:7862
To create a public link, set `share=True` in `launch()`.
Folder 100_SophiaCollegeLectureFace: 2100 steps
max_train_steps = 2100
stop_text_encoder_training = 0
lr_warmup_steps = 210
accelerate launch --num_cpu_threads_per_process=2 "train_network.py" --enable_bucket --pretrained_model_name_or_path="D:/AI Models/comfyui/checkpoints/ProtoGen_X3.4.safetensors" --train_data_dir="D:/AI Training/SophiaCollegeLectureFace/image" --resolution=512,512 --output_dir="D:/AI Training/SophiaCollegeLectureFace/model" --logging_dir="D:/AI Training/SophiaCollegeLectureFace/log" --network_alpha="128" --save_model_as=safetensors --network_module=networks.lora --text_encoder_lr=5e-5 --unet_lr=0.0001 --network_dim=128 --output_name="last" --lr_scheduler_num_cycles="1" --learning_rate="0.0001" --lr_scheduler="constant" --lr_warmup_steps="210" --train_batch_size="1" --max_train_steps="2100" --save_every_n_epochs="1" --mixed_precision="bf16" --save_precision="bf16" --caption_extension=".txt" --cache_latents --optimizer_type="AdamW8bit" --bucket_reso_steps=64 --xformers --bucket_no_upscale
And, once it is all over, you have your model in the "model" folder. Here is my LORA + training data as an example:
You must be registered to see the links
You can just copy the .safetensors file into webui/models/lora, and then you can load it in webui with the "additional networks" button. It should insert something into your prompt like <SophiaCollegeLectureFace:1> where the 1 is a weight. I found 1 is way too much, maybe like 0.8 at most. Otherwise it resulted in kind of ugly pictures.
And if you want a youtube video instead, use this:
You must be registered to see the links
that's where I got my information from mostly.
