support generating variations

[bakkot]

This is a continuation of #184 (and replaces that PR). I've credited @xraxra in the commit.

It adds two new flags:

• -v 0.1, which generates variations with a specific strength (as in the above PR)
• -V 1437160019,0.1;873874780,0.05, which takes a list of seed,weight pairs used to apply variations (and must be combined with --seed)

You use the first flag to generate variations, and the second to reproduce them - including as input for further variations; this supports creating variations of variations, so you can wander around the space in multiple steps.

Also, I've extended it to support img2img mode, so that you can use the same flow as with txt2img: run with prompt --init_image=whatever.png -n 10 to generate ten initial candidates, pick the one you like best, then generate ten variations of that one by doing prompt --init_image=whatever.png --seed=NNNNN -v 0.1 -n 10 (and, since you can do variations-of-variations, iterate as many times as you like).

Variations contain in their metadata (and stdout/dream_log.txt) the full command necessary to reproduce them precisely (assuming you use the same model/embeddings/init_img/etc, of course).

This doesn't support the web UI. Something to work on in a later PR. Also, the logic for generating metadata in dream.py is getting a bit convoluted, but that's something I'd like to work on anyway, again in a later PR.

To reproduce the --target_seed functionality from the original PR, you can use the -V switch: -v 0.1 -V NNNNN from the original PR is equivalent to -V NNNNN,0.1 in this PR.

[morganavr]

How the prompt must look to create a variation of the variation? This way?
"cute dog" -S 777777777 -v 0.1 -V 1437160019,0.1;873874780,0.05

If yes - then it's much more complex to use than in my version of code (where a tensor file is saved next to image).
In my version I can just type:
"cute dog" -S 777777777 -v 0.1
and software will automatically read all necessary data from file 777777777 .vtensor

If creating variations is indeed so complex (from end users POV) in this PR I propose to automatically read metadata from PNG file to free the users from typing unnecessary details.

[bakkot]

How the prompt must look to create a variation of the variation? This way?

Yes, that's what it looks like. You don't need to type it out by hand, though; the log message will print exactly the command you need to run to recreate the image, so you just copy-paste that and add -v 0.1 (or whatever) to the end. So from the user's POV there's no particular complexity involved.

I agree it would be nice to also support reading metadata from an existing image to create variations, but that can happen in a followup.

[bakkot]

@lstein I've rebased this and changed it to target the development branch. It's ready to go, I think.

[lstein]

I'm testing this out now. Is there any reason that the syntax for --with_variations can't be seed:weight,seed:weight, rather than seed,weight;seed,weight? The former would be more consistent with how we do weighting in the prompt. I care more about the colon than the semicolon.

[bakkot]

Is there any reason that the syntax for --with_variations can't be seed:weight,seed:weight, rather than seed,weight;seed,weight?

Nope, that should work fine. I was thinking of these as lists-of-lists and I reach for semicolons in that case, but you're right that colons would be better to match prompt weighting. I can fix that up if you'd like, or you can do it as you merge.

[lstein]

I'm about to knock off for the day. In a couple minutes I'll commit the PR to development, along with a little walkthrough/cheat-sheet that I wrote as I was testing. If you don't mind doing a new PR off of dev which swaps the commas for colons and the semicolons for commas, that would be great. There also need to be corresponding changes to the normalized prompt generator and probably the accompanying help text. I'm a little uncertain about the extended syntax for -V when you combine variation weighting with variation strength and don't want to meddle with that.

This is a truly wonderful feature! I could play with it all day. I bet we could do something amazing with the web interface too, like dragging and dropping one image on top of another.

[lstein]

I'm very happy to see this go in.

[magnusviri]

How does this feature work? Are there examples?

[bakkot]

@magnusviri There's a helpful readme written by @lstein. You can find some more examples of its outputs here or here (second link is a different repo but it works the same way).

[morganavr]

I have note tested this feature yet, just wanted to confirm another use case.
To create a variant of variant of variant... (10 times repeat word "variant") do I need to include all seeds and weights into the command line?

dream> "prompt" -S3357757885 -V3647897225,0.1;1614299449,0.02;-V3647347225,0.15;;-V364756225,0.3;;-V32447897225,0.0055;;-V3647897225,0.003;;-V3647897225,0.0038;;-V3647897225,0.0018; -v0.05 -n6

This prompt in unreal to construct. Why not store next to each image a .variants file with all necessary info to recreate it just from a seed? like so: "prompt" -S3357757885 (or inside EXIF metadata).

[bakkot]

To create a variant of variant of variant... (10 times repeat word "variant") do I need to include all seeds and weights into the command line?

Yes.

This prompt in unreal to construct.

You aren't expected construct it; you just copy-paste it from the log message or the image metadata.

Why not store next to each image a .variants file with all necessary info to recreate it just from a seed?

Well, to each their own, but personally I would find having to keep track of an additional file much, much more annoying than simply copy-pasting the log output, especially when trying to share things.