Skip to content

Not Issue. To whom may concern. You can quantize granite 4.1 on this version #18

@ed-hch

Description

@ed-hch

You can start from unsloth's BF16 quantized model and quantize to Q4_K_4 or Q8R16 following the instruction. And 30b can barley run on 24GB RAM in Q4_K_4. But slow of course.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions