LoRA: Custom layer filters that support regular expressions

[tehybel]

Old-style literal filters are still supported, since they are valid regular expressions, too.

[Nerogar]

Using regular expressions without any additional checks looks a bit dangerous. For example, the . character has a special meaning in regular expressions but is a very common character in module names. A filter like a.b could match unintended modules, not just those that contain exactly a.b

[dxqb]

• printing to the console which layers were selected (and which not) would be helpful imho
• when this is done I might want to use it for a PR to filter layers in full finetuning. Finetuning some layers such as the embedders seems counterproductive

[O-J1]

Given Thomas has been MIA for awhile now I have gone and refactored it.

@dxqbYD Added logging of the selected/unselected layers/

[dxqb]

Here

OneTrainer/modules/modelSetup/ChromaFineTuneSetup.py

Line 49 in d472c65

filtered_parameters = [param[1] for param in model.transformer.named_parameters() if "guidance_layer" not in param[0]]

is now code that applies a layer filter to full tinetuning - although technically requires_grad should also set to False for these parameters

[dxqb]

one possible solution for the some of the complications above:
do we really need regex? why not a simpler substring matching, and an "ignore pattern" that allows to exclude certain layers

[Calamdor]

I think that would allow filtering of “single” for the flux-ish models (flux, Hunyuan, i assume Chroma) to only grab a double block(s).

[dxqb]

I think that would allow filtering of “single” for the flux-ish models (flux, Hunyuan, i assume Chroma) to only grab a double block(s).

yes. any other use cases that would really require regex?

[O-J1]

I think that would allow filtering of “single” for the flux-ish models (flux, Hunyuan, i assume Chroma) to only grab a double block(s).

yes. any other use cases that would really require regex?

I think that would allow filtering of “single” for the flux-ish models (flux, Hunyuan, i assume Chroma) to only grab a double block(s).

yes. any other use cases that would really require regex?

Anyone wanting to to select multiple non-contiguous blocks or complex layer selection. Comma delimited lists still work fine and that was the intention of this

[dxqb]

yes. any other use cases that would really require regex?

Anyone wanting to to select multiple non-contiguous blocks or complex layer selection. Comma delimited lists still work fine and that was the intention of this

okay, if we want to support regex let's do that for expert users, but following the same sentiment as your screenshot quote I'd like a way to reject a pattern for non-expert users. I'd consider myself among those non-expert users, considering how difficult it is to reject something using regex:

^(?:(?!guidance_layer|context_embedder|x_embed).)*$

What do you think about:

• keep the current lora_layers (but rename it to layers_accept either in this PR or in a later one, because we also need it for FFT)
• add a [lora_]layers_reject for simple substring pattern matching to reject layers
• both can be used as regex if you switch it on

This solves the complexity, but also this #778 (comment) because we don't need regex presets anymore.

[O-J1]

This solves the complexity, ... don't need regex presets anymore.

Sumarising convo on Discord:

1. Regex presets are already supported
2. We agreed substring matching would be better as the default, to be contributed in a seperate PR (new UI field)

I believe all review comments have been addressed now or are relevant to a new PR