Feature Request: Hardware support check

[mdegans]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

Add a way to check if a model is supported on a given hardware backend.

Motivation

Currently, when an unsupported quantization is encountered, at least on Metal, asserts fire, leading to a crash. At least on my machine, called from a Rust GUI frontend, this manages to lock up the whole machine. Finder is unresponsive. Can't force quit. A cold reboot is actually necessary.

I don't believe there's a way to catch the assert so I'm looking a way to check for support and fail gracefully in such cases.

Possible Implementation

Here I'd be looking for advice but my initial thoughts would be to use the existing ggml_supports functions to check if the model's tensors are supported.

I see llama_model has some ggml_tensor and a std::vector of llama_layer also containing tensors. I figure there may be more to it than that, so I'd need guidance here.

The function might return a bool if the model is supported or some status code.

Alternatively, what would solve my immediate issue is for ggml_metal_graph_compute to return an error code in the cases where a model is unsupported. I see it already returns a ggml_status where GGML_STATUS_FAILED indicates failure. I'm assuming there's a good reason this behavior isn't already the case.

I'd be happy to work on and contribute such a feature if it would be of use.

[slaren]

If the supports_op function returns false on these operations, they will be automatically run on the CPU backend. With metal this should be fairly efficient because metal buffers can be used on the CPU backend without copies.

[mdegans]

if the supports_op function returns false on these operations, they will be automatically run on the CPU backend.

This doesn't seem to be happening because I am getting:

Asserting on type 30
GGML_ASSERT: /Users/.../.cargo/registry/src/index.crates.io-6f17d22bba15001f/llama-cpp-sys-3-0.5.9/external/llama.cpp/ggml-metal.m:1779: false && "not implemented"

30 turns out to be GGML_TYPE_BF16. The particular model was quantized with ./quantize --leave-output-tensor {name} 25 8

It seems to only be an issue with models quantized with this flag. I'm running the debugger and it seems the backend is indeed the CPU but am still reaching this assert.

[slaren]

Yes, because the ggml_metal_supports_op incorrectly returns true for this case. If you update this function to check for GGML_TYPE_BF16 on src[0] for MUL_MAT, it will be run on the CPU.

[mdegans]

Yes, because the ggml_metal_supports_op incorrectly returns true for this case. If you update this function to check for GGML_TYPE_BF16 on src[0] for MUL_MAT, it will be run on the CPU.

Thank you! I will do so and submit a PR if that works for you.

[slaren]

Sure, a PR to fix this would be welcome.

[mdegans]

Is BF16 is entirely unsupported for all operations on Metal?

[slaren]

Yes. As far as I know, BF16 is only supported on the CPU.

[mdegans]

Ok. In that case, mind if I put this...

    if (op->src[0]->type == GGML_TYPE_BF16) {
        return false;
    }

Above the switch? I'm assuming there's guaranteed to be at least one operand?

[slaren]

It's not guaranteed, eg. with GGML_OP_NONE. You should check if src[0] is NULL too.

[mdegans]

Got it! Will do! Something like this?

    for (size_t i = 0, n = 3; i < n; ++i) {
        if (op->src[i] != NULL && op->src[i]->type == GGML_TYPE_BF16) {
            return false;
        }
    }

Or is it unnecessary to check for anything but the first?

[slaren]

That should work. There may be a BF16 weight in src[1] in some ops.

[mdegans]

That should work. There may be a BF16 weight in src[1] in some ops.

Ok. The PR is submitted. If I manage to trigger any more asserts, I'll know where to go. Thanks again for your help!