<aside> 🚧 Full discolsure ! I am not an expert in training and fine-tuning models at all. I’ve barely trained under 10 LoRAs in my spare time, and done a handful of fine-tunes. Training is 100% something you need to get a grasp of. To really understand how it works, you need to experiment, experiment, experiment. Each concept/subject is different and reacts to different settings, and it’s difficult to understand which settings play significant roles during training. What I’m trying to say is — don’t take my word for it, also do your own research !
</aside>
When working with Stable Diffusion, you might encounter a fundamental issue. A certain type of images, or a certain object was under-represented during training, or straight up missing from the original dataset, and Stable Diffusion now struggles to generate quality images of that style/concept/object.
This can also be the case with specific image-treatments that are too niche to have been captioned correctly before training, or simply too niche to exist sufficiently in the base training dataset of Stable Diffusion, like thermal imagery, or the schlieren effect, to name a few examples.
Trying to generate an image of a night vision image of a man sitting on a bench yields the following type of images :
Not exactly the round cutout, grainy military type NVG image I had in mind
When faced with that situation, you can actually train Stable Diffusion, giving it more targeted training material so that I can generate exactly the type of images I want.