caption_separator cannot be specified in dataset config toml

[no-username-ideas-left]

When I try to use caption_separator in the dataset config toml, I get an error

  File "E:\ai_experimentation\kohya_ss\venv\lib\site-packages\voluptuous\schema_builder.py", line 433, in validate_mapping
    raise er.MultipleInvalid(errors)
voluptuous.error.MultipleInvalid: extra keys not allowed @ data['datasets'][0]['subsets'][0]['caption_separator']

example toml:

[[datasets]]
shuffle_caption = true

  [[datasets.subsets]]
  image_dir = 'E:\test_image\3_test_not_flip'
  flip_aug = false
  num_repeats = 3
  keep_tokens_separator = '|||'
  secondary_separator = ';;;'
  caption_separator = '. '

[rockerBOO]

Are you setting the caption_separator in other areas or is this all of it? This error is implying you are setting it multiple times, I believe.

[no-username-ideas-left]

I am using kohya_ss, so I guess it's possible it's also set there in some parameter. I didn't select it there and I don't see any option there to configure that value, so if it is set there, it's a default parameter.

For the error though, the documentation has this to say:

voluptuous.error.MultipleInvalid: extra keys not allowed @ ...: This error occurs when there is an option name that is not supported. It is highly likely that you misspelled the option name or mistakenly included it.

The other voloptuous errors mentioned all have "MultipleInvalid", so I don't think it necessarily indicates a key has been specified multiple times:

• voluptuous.error.MultipleInvalid: required key not provided @ ...: This error occurs when a required option is not provided. It is highly likely that you forgot to specify the option or misspelled the option name.
  • The error location is indicated by ... in the error message. For example, if you encounter an error like voluptuous.error.MultipleInvalid: required key not provided @ data['datasets'][0]['subsets'][0]['image_dir'], it means that the image_dir option does not exist in the 0th subsets of the 0th datasets setting.
• voluptuous.error.MultipleInvalid: expected int for dictionary value @ ...: This error occurs when the specified value format is incorrect. It is highly likely that the value format is incorrect. The int part changes depending on the target option. The example configurations in this README may be helpful.
• voluptuous.error.MultipleInvalid: extra keys not allowed @ ...: This error occurs when there is an option name that is not supported. It is highly likely that you misspelled the option name or mistakenly included it.

I think the error class is just for storing multiple errors:

class MultipleInvalid(Invalid):
    def __init__(self, errors=None):
        self.errors = errors[:] if errors else []

    def __repr__(self):
        return 'MultipleInvalid(%r)' % self.errors

[rockerBOO]

Second consideration may be that you are using the recent enough version that has this addition.

[no-username-ideas-left]

I'm using kohya_ss's latest version, which uses https://github.com/kohya-ss/sd-scripts/tree/bfb352bc433326a77aca3124248331eb60c49e8c , which is the latest 0.8.7 release

[rockerBOO]

So maybe something else I'm not considering. The code that accounts for this seems to be accurate. https://github.com/kohya-ss/sd-scripts/blob/main/library/config_util.py#L65

[rockerBOO]

Hmm seems it isn't in this part though https://github.com/kohya-ss/sd-scripts/blob/main/library/config_util.py#L184-L199 so maybe it is necessary to add here. If you add "caption_separator": str, to that list and see if that works to fix it.

[no-username-ideas-left]

Looks like that was the issue. Adding that line worked. Thanks!