PR to allow more user-defined optimizers by import path (package.module.name) without breaking prior configs. So far only works in advanced editor. Examples include prodigyplus.prodigy_plus_schedulefree.ProdigyPlusScheduleFree

[Epispasm]

This PR allows users to enter any optimizer import path into their config files. Prior to this PR, users could only enter the names hardcoded optimizers. There were only a few to pick from. This will open it up to any optimizer without needing to rewrite code or make major changes to config files.

I made one very small change to one single line in ai-toolkit/jobs/process/BaseSDTrainProcess.py - this script changes the case of the optimizer input to lower case, but this is redundant as your ai-toolkit/toolkit/optimzer.py also does this when comparing strings with names of hardcoded optimizers.

I tweaked the optimizer.py script to allow users to continue using the hard-coded names of optimizers - this will prevent existing configs to be broken. I added some changes at the end of the script which will dynamicailly load the entered optimizer.

No code changes needed to be made for optimizer_params as these are passed through from the config file when creating the optimizer.

Note, optimizers should throw their own errors, so you do not need to hardcode any error messages.

Examples:
bitsandbytes.optim.PagedAdamW
prodigyplus.prodigy_plus_schedulefree.ProdigyPlusScheduleFree
bitsandbytes.optim.LAMB