Eval bug: llama-server tool_calls returns arguments as JSON object instead of string, breaking OpenAI compatibility

[alvis233]

Name and Version

build: 8233 (c5a7788) with GNU 11.4.0 for Linux x86_64

Operating systems

Linux

GGML backends

CUDA

Hardware

NVIDIA GeForce RTX 5090

Models

• Qwen3-Coder-Next-UD-Q4_K_XL.gguf
  https://huggingface.co/unsloth/Qwen3-Coder-Next-GGUF

• Qwen3.5-35B-A3B-UD-Q4_K_XL.gguf
  https://huggingface.co/unsloth/Qwen3.5-35B-A3B-GGUF

Problem description & steps to reproduce

Dear mods, I am trying to run quantized model in llama.cpp through the instructions. However, following the recent Autoparser refactoring PR (#18675), llama-server returns the arguments field in tool_calls as a parsed JSON object rather than a JSON string.

This breaks strict OpenAI API compatibility. According to the OpenAI API Reference, tool_calls[].function.arguments must be a string containing JSON, not a parsed object. Because of this change, the official openai Python SDK (which uses Pydantic for strong type checking) crashes with a TypeError when attempting to process tool calls.

What I tried
Adding the argument --jinja or --chat-template chatml both failed to adhere to OpenAI API compatibility, which made the *claw frameworks (openclaw, nanoclaw, zeroclaw, ironclaw) fail to work as expected.

To Reproduce

1. Start llama-server with a tool-capable model (e.g., Qwen3-Coder-Next) and Jinja template enabled:

   ./llama-server --model Qwen3-Coder-Next-UD-Q4_K_XL.gguf --jinja --port 8001

2. Send a curl request to the /v1/chat/completions endpoint with tools provided:

   curl -s http://127.0.0.1:8001/v1/chat/completions \
     -H "Content-Type: application/json" \
     -d '{
       "model": "Qwen3-Coder-Next",
       "messages": [{"role": "user", "content": "What is 1+1? Use the add tool."}],
       "tools": [{"type": "function", "function": {"name": "add", "description": "Add two numbers", "parameters": {"type": "object", "properties": {"a": {"type": "string"}, "b": {"type": "string"}}, "required": ["a", "b"]}}}],
       "tool_choice": "auto"
     }'

3. Observe the tool_calls block in the raw JSON response. It shows:

   "tool_calls": [
     {
       "type": "function",
       "function": {
         "name": "add",
         "arguments": {
           "a": "1",
           "b": "1"
         }
       },
       "id": "..."
     }
   ]

   Instead of the expected OpenAI-compatible format:

   "arguments": "{\"a\": \"1\", \"b\": \"1\"}"

4. If you run the official openai Python SDK (v2.21.0), it immediately crashes upon receiving the tool call response:

   TypeError: the JSON object must be str, bytes or bytearray, not dict

Root Cause
I traced this back to the massive parser refactoring in PR #18675 (commit 566059a26b0ce8faec4ea053605719d399c64cc5).
In common/chat.cpp around line 132, the arguments field is explicitly parsed into a JSON object:

{"type", "function"},
{"function", {
    {"name", tool_call.name},
    {"arguments", json::parse(tool_call.arguments)}, // <-- This causes the issue
}},

It should output the raw serialized JSON string instead of parsing it.

First Bad Commit

566059a
(From PR #18675: Autoparser - complete refactoring of parser architecture)

Relevant log output

Logs

$ ./llama.cpp/build/bin/llama-server \
    --model /home/alvis/Workspace/MachineLearning/Data/LinkedData/Qwen3-Coder-Next-UD-Q4_K_XL.gguf \
    --alias "Qwen3-Coder-Next" \
    --ctx-size 16384 \
    --seed 3407 \
    --temp 1.0 \
    --top-p 0.95 \
    --min-p 0.01 \
    --top-k 40 \
    --host 0.0.0.0 \
    --port 8001 \
    --jinja

ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 5090, compute capability 12.0, VMM: yes
main: n_parallel is set to auto, using n_parallel = 4 and kv_unified = true
build: 8233 (c5a778891) with GNU 11.4.0 for Linux x86_64
system info: n_threads = 6, n_threads_batch = 6, total_threads = 20

system_info: n_threads = 6 (n_threads_batch = 6) / 20 | CUDA : ARCHS = 750,800,860,890,1200,1210 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | BLACKWELL_NATIVE_FP4 = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX_VNNI = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | 

init: using 19 threads for HTTP server
start: binding port with default address family
main: loading model
...
main: server is listening on http://0.0.0.0:8001
main: starting the main loop...

# Then triggered the API request:
$ curl -s http://127.0.0.1:8001/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen3-Coder-Next",
    "messages": [{"role": "user", "content": "What is 1+1? Use the add tool."}],
    "tools": [{"type": "function", "function": {"name": "add", "description": "Add two numbers", "parameters": {"type": "object", "properties": {"a": {"type": "string"}, "b": {"type": "string"}}, "required": ["a", "b"]}}}],
    "tool_choice": "auto"
  }' | python3 -m json.tool

# Server output:
srv  update_slots: all slots are idle
srv  params_from_: Chat format: peg-native
slot get_availabl: id  3 | task -1 | selected slot by LRU, t_last = -1
slot launch_slot_: id  3 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> min-p -> ?xtc -> ?temp-ext -> dist 
slot launch_slot_: id  3 | task 0 | processing task, is_child = 0
slot update_slots: id  3 | task 0 | new prompt, n_ctx_slot = 16384, n_keep = 0, task.n_tokens = 288
slot init_sampler: id  3 | task 0 | init sampler, took 0.04 ms, tokens: text = 288, total = 288
slot update_slots: id  3 | task 0 | prompt processing done, n_tokens = 288, batch.n_tokens = 288
slot      release: id  3 | task 0 | stop processing: n_tokens = 311, truncated = 0
srv  update_slots: all slots are idle
srv  log_server_r: done request: POST /v1/chat/completions 127.0.0.1 200

# Received JSON response violating the string arguments spec:
{
    "choices": [
        {
            "finish_reason": "tool_calls",
            "index": 0,
            "message": {
                "role": "assistant",
                "content": "",
                "tool_calls": [
                    {
                        "type": "function",
                        "function": {
                            "name": "add",
                            "arguments": {
                                "a": "1",
                                "b": "1"
                            }
                        },
                        "id": "VR8m59fStegbYHZWeoJlj4nI0j9hhTXt"
                    }
                ]
            }
        }
    ],
    ...
}

[simonxluo]

the same problem.it may has something to do with c024d85

[tarruda]

Seems like the issue is related to constrained JSON output. I can reproduce it consistently with Qwen 3.5 4B:

llama-server --no-mmap --no-warmup --ctx-size 32768 --jinja -fa on --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.
0 --hf-repo unsloth/Qwen3.5-4B-GGUF:q4_k_m

And this node.js script:

const fs = require('fs');
const http = require('http');

// Configuration
const API_URL = 'http://127.0.0.1:8080/v1/chat/completions';
const MODEL = 'gpt';

// Define the JSON schema for response format
const jsonSchema = {
  strict: true,
  schema: {
    type: 'array',
    items: {
      type: 'object',
      properties: {
        key: { type: 'string' },
        value: { type: 'string' }
      },
      required: ['key', 'value'],
      additionalProperties: false
    }
  }
};

// Build request body
const requestBody = {
  model: MODEL,
  messages: [
    {
      role: 'system',
      content: [
        {
          type: 'text',
          text: 'Extract all key/values as a JSON array where each item has the format {key, value}'
        },
      ]
    },
    {
      role: 'user',
      content: [
        {
          type: 'text',
          text: 'name: rust, kind: programming language'
        },
      ]
    }
  ],
  response_format: {
    type: 'json_schema',
    json_schema: jsonSchema
  },
  chat_template_kwargs: {
    enable_thinking: false
  }
};

// Parse URL to extract hostname and port
const url = new URL(API_URL);
const options = {
  hostname: url.hostname,
  port: url.port || 80,
  path: url.pathname,
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'Content-Length': Buffer.byteLength(JSON.stringify(requestBody)),
  }
};

// Make HTTP request
const req = http.request(options, (res) => {
  let data = '';

  res.on('data', (chunk) => {
    data += chunk;
  });

  res.on('end', () => {
    try {
      const response = JSON.parse(data);
      // Extract and output only the text content
      const content = response.choices?.[0]?.message?.content;
      if (content) {
        console.log(content);
      } else {
        console.error('No content found in response. Response status:' + res.statusCode);
      }
    } catch (e) {
      console.error('Failed to parse response:', data);
    }
  });
});

req.on('error', (e) => {
  console.error('Request error:', e.message);
});

req.write(JSON.stringify(requestBody));
req.end();

The script returns 500.

It seems like a combination of things. For example, the chat_template_kwargs parameter is removed, seems to work fine. Actually, that doesn't seem consistent. I just tried without chat_template_kwargs and it failed too.

Server logs:

srv    operator(): got exception: {"error":{"code":500,"message":"Failed to parse input at pos 2122: ```json\n[\n  {\n    \"key\": \"name\",\n    \"value\": \"rust\"\n  },\n  {\n    \"key\": \"kind\",\n    \"value\": \"programming language\"\n  }\n]\n```","type":"server_error"}}

According to the server logs, it seems like the model is still being allowed to output fenced code blocks even though the output should have been constrained, and this causes some internal parsing error within llama.cpp

cc @pwilkin

[pwilkin]

This was a conscious decision. While the official API requires the arguments to be a string, in practice most clients expect it to be a JSON object. I know this is hard to balance since that's what the official API claims, but in practice, we're trying to adapt to what clients expect instead.

I'll add a switch to trigger this behavior.

The structured output is another issue, thanks for the reproduction code.

[tarruda]

Here's a script that makes it easier to reproduce (just appends an instruction to the system prompt):

const fs = require('fs');
const http = require('http');

// Configuration
const API_URL = 'http://mac.lan.internal:8080/v1/chat/completions';
const MODEL = 'gpt';

// Define the JSON schema for response format
const jsonSchema = {
  strict: true,
  schema: {
    type: 'array',
    items: {
      type: 'object',
      properties: {
        key: { type: 'string' },
        value: { type: 'string' }
      },
      required: ['key', 'value'],
      additionalProperties: false
    }
  }
};

// Build request body
const requestBody = {
  model: MODEL,
  messages: [
    {
      role: 'system',
      content: [
        {
          type: 'text',
          text: 'Extract all key/values as a JSON array where each item has the format {key, value}. Wrap the output in a fenced markdown code block'
        },
      ]
    },
    {
      role: 'user',
      content: [
        {
          type: 'text',
          text: 'name: rust, kind: programming language'
        },
      ]
    }
  ],
  response_format: {
    type: 'json_schema',
    json_schema: jsonSchema
  },
  chat_template_kwargs: {
    enable_thinking: false
  }
};

// Parse URL to extract hostname and port
const url = new URL(API_URL);
const options = {
  hostname: url.hostname,
  port: url.port || 80,
  path: url.pathname,
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'Content-Length': Buffer.byteLength(JSON.stringify(requestBody)),
  }
};

// Make HTTP request
const req = http.request(options, (res) => {
  let data = '';

  res.on('data', (chunk) => {
    data += chunk;
  });

  res.on('end', () => {
    try {
      const response = JSON.parse(data);
      // Extract and output only the text content
      const content = response.choices?.[0]?.message?.content;
      if (content) {
        console.log(content);
      } else {
        console.error('No content found in response. Response status:' + res.statusCode);
      }
    } catch (e) {
      console.error('Failed to parse response:', data);
    }
  });
});

req.on('error', (e) => {
  console.error('Request error:', e.message);
});

req.write(JSON.stringify(requestBody));
req.end();

Even with the 0.8 it becomes reproducible:

llama-server --no-mmap --no-warmup --ctx-size 32768 --jinja -fa on --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.0 --hf-repo unsloth/Qwen3.5-0.8B-GGUF:q8_0

[gsgxnet]

I am not sure if the error I see is really related to the issue opened here.

OpenAI codex in the llama.cpp git folder. Codex Version 0.111. Configured to use the Qwen3.5-122B-A10B model served by llama.cpp server, using responses API. llama.cpp version with last commit c5a77889

>> Code review started: changes against 'master' <<

⚠ Model metadata for `Qwen3.5-122B-A10B` not found. Defaulting to fallback metadata; this can degrade performance and cause issues.


› Review the code changes against the base branch 'master'. The merge base commit for this comparison is c5a778891ba0ddbd4cbb507c823f970595b1adc2. Run `git diff
  c5a778891ba0ddbd4cbb507c823f970595b1adc2` to inspect the changes relative to master. Provide prioritized, actionable findings.


■ {"error":{"code":400,"message":"Unable to generate parser for this template. Automatic parser generation failed: \n------------\nWhile executing CallExpression at line 145, column 28 in
source:\n... {%- else %}↵        {{- raise_exception('Unexpected message role.') }}↵    {%- ...\n                                           ^\nError: Jinja Exception: Unexpected message
role.","type":"invalid_request_error"}}

<< Code review finished >>

• Review was interrupted. Please re-run /review and wait for it to complete.

Using that same codex with an older llama-server running on an other Mac serving qwen3-coder-next-80b runs fine.

If you think my issue does not fit here, I will open a new issue. I could not find any other issue looking more similar to mine than this one here.

[pwilkin]

@gsgxnet The developer role is not supported. Since this is happening quite often, I think we'll just add a fallback for this.

[gsgxnet]

For reference this is the log of the same (fake) /review request, this time codex using the other llama-server (qwen3-coder-next-80b, no thinking)

>> Code review started: changes against 'master' <<

› Review the code changes against the base branch 'master'. The merge base commit for this comparison is c5a778891ba0ddbd4cbb507c823f970595b1adc2. Run `git diff
  c5a778891ba0ddbd4cbb507c823f970595b1adc2` to inspect the changes relative to master. Provide prioritized, actionable findings.


• Ran cd /Users/Shared/A/DEVE/LLAMACPP/llama-server.cpp && git diff c5a778891ba0ddbd4cbb507c823f970595b1adc2
  └ (no output)

─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

• I'll review the code changes against the base branch 'master'. First, let me get the diff to see what changes were made.

• Ran cd /Users/Shared/A/DEVE/LLAMACPP/llama-server.cpp && git status && echo "---" && git log --oneline -5
  └ On branch master
    Your branch is up to date with 'origin/master'.
    … +6 lines
    2f2923f89 Autoparser: add optional argument reshuffle capability (#20171)
    649f06481 quants : Add memsets and other fixes for IQ quants (#19861)

─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

• The diff command returned no output, which suggests there may be no changes or the diff might be empty. Let me check the current branch and status to understand the repository state.

• Ran cd /Users/Shared/A/DEVE/LLAMACPP/llama-server.cpp && git diff c5a778891ba0ddbd4cbb507c823f970595b1adc2..HEAD
  └ (no output)

─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

• I see we're on the master branch and the working tree is clean. Let me check if there are any uncommitted changes or if the diff is between master and itself:

• Ran cd /Users/Shared/A/DEVE/LLAMACPP/llama-server.cpp && git rev-parse HEAD
  └ c5a778891ba0ddbd4cbb507c823f970595b1adc2

─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

• The diff is empty, which means HEAD is at the same commit as the merge base (c5a778891). There are no changes to review. Let me verify this:

  There are no changes to review - the current HEAD is at the same commit as the merge base, meaning no PR changes have been made.

[Atomic-Germ]

Actually, a lot of the MCP implementation is broken. By broken I mean going against the way every MCP is coded, despite standards being pretty clear by this point.

[ZUIcat]

I just updated my tool yesterday to work with JSON objects, but it looks like I'll need to revert the changes...

[pwilkin]

Sorry for that, you can support both just in case :)

[CISC]

Sorry for that, you can support both just in case :)

TBH, all tools should support both, never understood the reasoning behind OAI requiring string there.

[gsgxnet]

I have given the new llama-server which includes b283f6d5b Revert to OAI-compatible args (#20213) (build: 8240 (d088d5b74) with AppleClang 17.0.0.17000013 for Darwin arm64) a try.

Same as before. Within the llama.cpp git repo call codex to do a (fake) /review. But still get the same error:

>> Code review started: changes against 'master' <<

⚠ Model metadata for `Qwen3.5-122B-A10B` not found. Defaulting to fallback metadata; this can degrade performance and cause issues.


› Review the code changes against the base branch 'master'. The merge base commit for this comparison is d088d5b74f1d63b9a345d1515ab9e3bb3bc81a10. Run `git diff
  d088d5b74f1d63b9a345d1515ab9e3bb3bc81a10` to inspect the changes relative to master. Provide prioritized, actionable findings.


■ {"error":{"code":400,"message":"Unable to generate parser for this template. Automatic parser generation failed: \n------------\nWhile executing CallExpression at line 145, column 28 in
source:\n... {%- else %}↵        {{- raise_exception('Unexpected message role.') }}↵    {%- ...\n                                           ^\nError: Jinja Exception: Unexpected message
role.","type":"invalid_request_error"}}

May be this is really a different issue than the issue starter sees (saw?). I have configured codex to use the OpenAI </v1/responses> API, not the completions.