feat: Add Bitsandbytes quantization for transformer backend enhancement #1775 and fix: Transformer backend error on CUDA #1774

[fakezeta]

Add BitsAndBytes Quantization and fixes embedding on CUDA devices

This PR fixes #1775 and #1774

Notes for Reviewers
LowVRAM toggles INT4 quantization
F16Memory toggles bfloat16 compute type instead of float32

This is my first PR so be kind :)

Signed commits

• Yes, I signed my commits.

[netlify]

✅ Deploy Preview for localai canceled.

Name	Link
🔨 Latest commit	d304e33
🔍 Latest deploy log	https://app.netlify.com/sites/localai/deploys/65f33abcb3c03d0008073e92

[mudler]

as we do actually have a quantization field in the gRPC requests and BitsAndBytes is one of the available ( see https://huggingface.co/docs/transformers/main_classes/quantization#quantization), we might want to select BitsAndBytesConfig when setting quantization=BitsAndBytesConfig instead? so we keep the backward compatibility change too (this is more of a question).

Otherwise changes looks good to me, thanks @fakezeta !

[fakezeta]

Thank you @mudler I think it's a good point.

What do you think of using something like:

• quantization=bnb_4bit
• quantization=bnb_8bit

to select between the two options?

[mudler]

@fakezeta that looks reasonable to me 👍

[mudler]

Looks good here, however CI seems to fail: https://github.com/mudler/LocalAI/actions/runs/8282698873/job/22664377430?pr=1823

[fakezeta]

I was looking at the logs, the problem is related to non-cuda build.
Now test.sh is working also on non-cuda build and without bitsandbytes installed.

[mudler]

Looking good, thanks @fakezeta !