Add llmaz as another platform to run llama.cpp on Kubernetes

[kerthcet]

• I have read the contributing guidelines
• Self-reported review complexity:
  • Low
  • Medium
  • High

Hi, llmaz is a platform to serve large language models on Kubernetes, llama.cpp is an vital part of it for CPU inference as well as GPU part. Here's an example:

apiVersion: llmaz.io/v1alpha1
kind: OpenModel
metadata:
  name: qwen2-0-5b-gguf
spec:
  familyName: qwen2
  source:
    modelHub:
      modelID: Qwen/Qwen2-0.5B-Instruct-GGUF
      filename: qwen2-0_5b-instruct-q5_k_m.gguf

apiVersion: inference.llmaz.io/v1alpha1
kind: Playground
metadata:
  name: qwen2-0-5b
spec:
  replicas: 1
  modelClaim:
    modelName: qwen2-0-5b-gguf
  backendConfig:
    name: llamacpp
    args:
    - -fa # use flash attention

This is all your need to do, then you can serve models with llama.cpp on Kubernetes.

Thanks!

[kerthcet]

kindly ping @ggerganov

[kerthcet]

Hi @ggerganov let me know if this is a suitable integration for llama.cpp, actually, we use llama.cpp a lot in our platform and we use it for all kinds of tests as well for cost saving. I believe it's a great showcase about llama.cpp in cloud. Thanks!