Model loading from huggingface using parallel downloads

[dxqb]

• increases download speed from about 40 mbit/s to the maximum download limit of my connection, by utilizing the parallel downloads feature of the huggingface_hub API
• works on any host including your local machine, but especially useful for cloud training, because the model is downloaded anew each training run (if you don't have persistent storage)
• implemented only for Flux, hence the "Draft PR". Will copy to all models if you like this PR

[dxqb]

merged #639 because there is a small dependency. Find the stand-alone code of this PR here: 851c3f2

[dxqb]

merged the new master branch, which makes this PR easier to read because it was a combination before

[dxqb]

Please review this when you have the time, even though it is still a draft, because the only open point is to copy it to all models.
It is a big time and cost saver on clouds, cutting down model download time from ~ 15 minutes to ~ 3 minutes.

[dxqb]

• copy to other models, at least the newer and larges ones

then ready for merge

[O-J1]

For future maintainers/ref:

HF_HUB_ENABLE_HF_TRANSFER is disabled by default. Dont ever enable it without a toggle. This PR does not enable it.

Since huggingface_hub > 32.0.0, hf-xet capabilities were merged into the base package. Xet is buggy and not ready for consumer usage. We need to set HF_HUB_DISABLE_XET=1. This will slow downloads down but that does not matter for consumer systems, you only download once.

If end users want it, they need to make manual modifications themselves but given the widespread reports its not usable at this current time.

Test results

Worked without issue on 2 runs but given my findings in their docs and repo, its only ready for review (and then merge) after:

• Ensure other large models utilise huggingface.snapshot_download
• we modify start-ui.bat and sh to launch with HF_HUB_DISABLE_XET=1 set.

start-ui.bat suggested modification:

REM Ensure HF_HUB_DISABLE_XET as it's still buggy; default disables XET (set to 0 to enable)
if not defined HF_HUB_DISABLE_XET (
    set "HF_HUB_DISABLE_XET=1"
)
echo HF_HUB_DISABLE_XET=%HF_HUB_DISABLE_XET%
echo.
echo NOTE: XET (when enabled) allows higher speed parallel downloads which can increase throughput, however it is buggy.
echo NOTE: Only enable XET if your download speed is greater than 40 megabytes per second (MB/s) — not megabits.
echo To enable XET, either export HF_HUB_DISABLE_XET=0 in the venv or modify start-ui.bat directly before running (the latter will break git fetch).
echo.

:launch
...

References on why we have to disable it on consumer systems:

huggingface/xet-core#446
huggingface/xet-core#409
huggingface/xet-core#448
huggingface/xet-core#400
Multiple reports in discord (HF official discord and others)

[dxqb]

tested on all implemented models