feat: Add Bitsandbytes quantization for transformer backend

[fakezeta]

Is your feature request related to a problem? Please describe.

Quantization is not available for transformer backend
Describe the solution you'd like

Add bitsandbytes 4bit quantization to be triggered by the user with the low_vram flag in the model definition
Additionally I propose to use the f16 flag to change the compute_dtype to bfloat16 for better optimization on Nvidia cards
Describe alternatives you've considered

Additional context

I've implemented this while fixing #1774
Issue opened for tracking

[mudler]

Is your feature request related to a problem? Please describe.

Quantization is not available for transformer backend Describe the solution you'd like

Add bitsandbytes 4bit quantization to be triggered by the user with the low_vram flag in the model definition Additionally I propose to use the f16 flag to change the compute_dtype to bfloat16 for better optimization on Nvidia cards Describe alternatives you've considered

Additional context

I've implemented this while fixing #1774 Issue opened for tracking

good point @fakezeta! do you have already have the changes and come up with a PR? maybe we can take it from there

[fakezeta]

You know I'm old, give me some time 😄
I think I'll do it tonight (CET).

[mudler]

You know I'm old, give me some time 😄 I think I'll do it tonight (CET).

That's fine, actually, thanks for the efforts!