Qwen3 Reranker: rankAndSort() returns NaN/null scores

[jesper-bylund]

Bug Description

LlamaRankingContext.rank() returns NaN and rankAndSort() returns null scores for all documents when using the Qwen3 Reranker model (ggml-org/Qwen3-Reranker-0.6B-Q8_0-GGUF).

Root Cause

The Qwen3 Reranker uses a thinking-based reranking template that expects the model to generate tokens (think + yes/no answer), not produce a single embedding logit:

<|im_start|>system
Judge whether the Document meets the requirements based on the Query and the Instruct provided. Note that the answer can only be "yes" or "no".<|im_end|>
<|im_start|>user
<Instruct>: Given a web search query, retrieve relevant passages that answer the query
<Query>: {query}
<Document>: {document}<|im_end|>
<|im_start|>assistant
<think>

</think>

However, _evaluateRankingForInput() in LlamaRankingContext only runs one token and reads the embedding logit:

const embedding = this._llamaContext._ctx.getEmbedding(input.length, 1);
const logit = embedding[0];
const probability = logitToSigmoid(logit);

This works for traditional cross-encoder rerankers (like jina-reranker-v2) that produce a relevance logit directly, but Qwen3's architecture requires generating the <think>...</think> reasoning and then a "yes"/"no" token, using the logprobs of "yes" vs "no" as the relevance score.

Reproduction

import { getLlama } from "node-llama-cpp";

const llama = await getLlama();
const model = await llama.loadModel({
  modelPath: "hf:ggml-org/Qwen3-Reranker-0.6B-Q8_0-GGUF/qwen3-reranker-0.6b-q8_0.gguf"
});

const ctx = await model.createRankingContext({ contextSize: 4096 });

const score = await ctx.rank("What are projects?", "Projects use NNN naming convention");
console.log(score); // NaN

const sorted = await ctx.rankAndSort("test query", ["relevant doc", "irrelevant doc"]);
console.log(sorted); // [{ document: "...", score: null }, { document: "...", score: null }]

await ctx.dispose();
await model.dispose();
await llama.dispose();

Expected Behavior

For Qwen3-style rerankers, the implementation should:

1. Generate tokens until the model produces "yes" or "no" (after the <think> block)
2. Use the logprob of the "yes" token as the relevance score (as described in the Qwen3 Reranker documentation)

Environment

• node-llama-cpp: 3.15.1
• Model: hf:ggml-org/Qwen3-Reranker-0.6B-Q8_0-GGUF/qwen3-reranker-0.6b-q8_0.gguf
• OS: macOS (Apple Silicon)
• Runtime: Bun 1.3.9

Related

• llama.cpp #16407 — similar broken scores via llama-server
• llama.cpp #17743 — empty output from Qwen3 reranker
• This model is the default reranker in QMD, affecting all QMD users' query command

[jesper-bylund]

Updated analysis — the root cause is clearer now

After digging into the llama.cpp PR #14029 that added Qwen3 reranker support, the issue is more specific than I initially described.

How traditional rerankers work (jina, bge)

These are non-causal cross-encoders with a pooling layer. Feed in query+document, read a single pooled relevance logit. getEmbedding(inputLength, 1) works perfectly for these.

How Qwen3 reranker works

It's a causal LM with a classification head — fundamentally different architecture. From the official Qwen3 usage code:

# The template pre-fills an empty <think></think> block — no generation needed
prefix = "<|im_start|>system\nJudge whether...<|im_end|>\n<|im_start|>user\n"
suffix = "<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n"

# Score = softmax over "yes" vs "no" token logits at the LAST position
batch_scores = model(**inputs).logits[:, -1, :]  # full vocab logits at last token
true_vector = batch_scores[:, token_true_id]      # logit for "yes"
false_vector = batch_scores[:, token_false_id]    # logit for "no"
batch_scores = torch.stack([false_vector, true_vector], dim=1)
batch_scores = torch.nn.functional.log_softmax(batch_scores, dim=1)
scores = batch_scores[:, 1].exp().tolist()

Key insight: no token generation happens. The <think>\n\n</think> is pre-filled in the input. You just need the raw vocabulary logits at the last position, then compare P("yes") vs P("no").

What _evaluateRankingForInput currently does

const embedding = this._llamaContext._ctx.getEmbedding(input.length, 1);
const logit = embedding[0];
const probability = logitToSigmoid(logit);

This reads a pooled embedding value, which is meaningful for traditional cross-encoders but returns garbage (leading to NaN) for Qwen3's causal LM + classification head architecture.

What it should do for Qwen3-style models

Instead of getEmbedding(), it needs to:

1. Get the full vocabulary logits at the last token position
2. Extract logits for the "yes" and "no" token IDs
3. Apply softmax over [no_logit, yes_logit]
4. Return the "yes" probability as the score

From the llama.cpp PR discussion

ngxson (the PR author) noted:

"The main problem is that qwen's rerank model uses causal attention, it's simply a normal next generation model which outputs either 'yes' or 'no' token"

ggerganov responded:

"For Qwen3 rerank, what you seem to need is to pool using last and apply the classification head on the result — the latter is missing"

Even in llama.cpp itself, this wasn't cleanly resolved — there are open issues (#16407, #17743) with broken scores. The pool using last + classification head approach was a workaround.

Suggested fix

Detection: check if chat_template.rerank exists and contains the Qwen3 "yes"/"no" judge pattern (or check model metadata for pooling_type: last + causal attention).

For these models, replace the getEmbedding() path with a logits-based scoring path that reads vocab logits at the last position and computes softmax("yes", "no").

[jesper-bylund]

Further investigation — the bug is deeper than the ranking context

What I tested

1. createRankingContext().rank() → NaN (as reported)
2. createContext() + evaluateWithMetadata() with probabilities: true → ALL 151K vocabulary probabilities are NaN
3. Simple prompt "Hello, how are you?" through regular generation → also all-NaN logits

The model loads successfully, supportsRanking: true, architecture is qwen3, but no meaningful logits are produced at all — not even for basic text generation.

Environment details

• node-llama-cpp 3.15.1 bundles llama.cpp b7836 (Jan 25, 2026)
• Qwen3-Reranker support was merged in PR #15824 (Sep 25, 2025) — so it's included
• Model metadata: hasDecoder: true, hasEncoder: false, no pooling type set
• The model has chat_template.rerank but no explicit pooling_type in GGUF metadata

Root cause hypothesis

The llama.cpp #15824 implementation of Qwen3 reranker support may have issues with how it handles this model's architecture. The original PR #14029 (which was closed in favor of #15824) discussed problems with:

• build_inp_cls() containing all token indices instead of just the output token
• The assumption that CLS/RANK models are non-causal (Qwen3 reranker IS causal)
• Missing classification head application after last-position pooling

This matches the open llama.cpp issues #16407 (scores of 1e-28) and #17743 (empty output) — the bug appears to be in llama.cpp's C++ implementation for this model architecture, and node-llama-cpp inherits it.

What this means for node-llama-cpp

Even if node-llama-cpp adds a logits-based scoring path (extracting yes/no token probabilities instead of using getEmbedding()), it won't work until the underlying llama.cpp evaluation produces valid logits for this model.

The fix likely needs to happen in llama.cpp first (or in how the GGUF conversion creates the model file), then node-llama-cpp will benefit automatically.

[jesper-bylund]

Root Cause Found — Bad GGUF Conversion, Not a Code Bug

After extensive debugging, the issue is not in node-llama-cpp — it's in the GGUF model file.

The Problem

The model hf:ggml-org/Qwen3-Reranker-0.6B-Q8_0-GGUF was converted using the automated "GGUF-my-repo" space, which used a conversion script that predates Qwen3 reranker support (PR ggml-org/llama.cpp#15824). This produces a GGUF that looks correct (has cls.output.weight, chat_template.rerank, pooling_type=4) but returns NaN from the C++ embedding layer.

We confirmed this by testing with llama-embedding CLI (llama.cpp b7990) — same NaN result:

$ llama-embedding -m hf_ggml-org_qwen3-reranker-0.6b-q8_0.gguf --embd-normalize -1 \
  -p "What is the capital of France?\tParis is the capital of France."
rerank score 0:      nan [yes]
rerank score 0:      nan [no]

The Fix

Using the correctly-converted model from giladgd/Qwen3-Reranker-0.6B-GGUF (the same one recommended in PR #506) works perfectly:

$ llama-embedding -m giladgd_Qwen3-Reranker-0.6B.Q8_0.gguf --embd-normalize -1 \
  -p "What is the capital of France?\tParis is the capital of France."
rerank score 0:    0.997 [yes]
rerank score 0:    0.003 [no]

And via node-llama-cpp rankAndSort():

const sorted = await ctx.rankAndSort("capital of France", [
  "Paris is the capital",
  "Bananas are yellow",
  "France is in Europe"
]);
// [{ document: "Paris is the capital", score: 0.729 },
//  { document: "France is in Europe", score: 0.500 },
//  { document: "Bananas are yellow", score: 0.500 }]

Minor Note

The scores from node-llama-cpp are compressed (0.729 instead of 0.997) because _evaluateRankingForInput applies sigmoid() to the embedding value, which for Qwen3 is already a probability (0-1). The sigmoid of 0.997 ≈ 0.729. The ranking order is preserved, so it works for sorting, but the absolute scores are less meaningful. This could be improved by skipping sigmoid when the model's pooling type is RANK and the output is already in [0,1] range.

Summary

• Not a node-llama-cpp bug — the code in PR feat: Qwen3 Reranker support #506 works correctly
• Root cause: ggml-org's auto-converted GGUF is broken for Qwen3 reranker
• Solution: Use hf:giladgd/Qwen3-Reranker-0.6B-GGUF:Q8_0 (as recommended in feat: Qwen3 Reranker support #506)
• Affected: Anyone using hf:ggml-org/Qwen3-Reranker-0.6B-Q8_0-GGUF (including QMD's default)

Closing as this is a model conversion issue, not a library bug.