[Feat]: Smarter caching instead of all or nothing - graceful fails, reduce repeated work, portable cache

[shawndolen]

Describe your use-case.

I have large-ish datasets. There have been multiple at home users over the past year reporting issues with 20k+ datasets where hours of caching can be can be cancelled by a single invalid image.

Additionally the caching is "all or nothing", starting from scratch when we only add a couple new images and adjust some captions. Additionally I would like to cache on my own machine and then transfer these files to rented machine. Have that cache be portable would be extremely useful. Especially with SD3.5 and Flux, offloading pushes dataloader threads down to 1, would be a quick way to burn money on a rented GPU doing that.

What would you like to see as a solution?

I would like to see:

• The caching be done differentially
• To gracefully fail in the event of a invalid image or caption
• That the cache be portable enabling users to cache on their home GPU, this would cheapen the cost to train loras and finetune for users as the slow part (caching) could be done at home.

Have you considered alternatives? List them here.

N/A

[gilga2024]

I can only second this one. I have to use a folder consisting out of several thousand pictures for several trainings and therefor it has to be cached again and again. This is time consuming and costs a lot of energy too.

Simple implementation idea:
I would like to have a "persistent concept cache" option + "create now"-button in the concept settings that allows me to cache the concept "outside" of the current workspace (e.g. using an identifier or folder to store it in a general concept-cache folder). The concept should then be imported, meaning copy all cached content, into the workspace each time the cache is build (in order to not break any other existing functionality).

This is not the "perfect" solution and differs a bit from what is described above. But at least relevant functionality would be present to build what is intended there (add a second small concept for the new pictures, also cache the new ones persistently + rebuild that small cache every time you change/add something new , old ones stay cached in the main concept; only rebuild the main cached concept once you are sure the new ones should stay in your main concept / reached an amount that justifies a complete rebuild of the main cache).

[DaStapo]

Was about to request the same thing; pretty annoying with 100k images.