How to get the performance metrics of KVzip in the graph?

[xhchen28]

I cloned the kvpress repository and ran the evaluate.sh script directly using the Qwen3-8B model. However, the results I obtained are significantly different from those presented in the graph on Hugging Face.

[xhchen28]

My setup details are below：
dataset: ruler
data_dir: 4096
model: models/Qwen3-8B
device: cuda:0
press_name: kvzip
compression_ratio: 0.5
key_channel_compression_ratio: null
fraction: 1.0
max_new_tokens: null
max_context_length: null
compress_questions: false
needle_depth: null
compression_interval: null
target_size: null
hidden_states_buffer_size: null
output_dir: ./results
log_level: INFO
model_kwargs:
attn_implementation: flash_attention_2
torch_dtype: auto
press_init_command: KVzipPress(compression_ratio=0.5, layerwise=False, n_sink=4)
seed: 42

My results details are below:
{
"cwe": {
"string_match": 62.58
},
"fwe": {
"string_match": 86.8
},
"niah_multikey_1": {
"string_match": 48.4
},
"niah_multikey_2": {
"string_match": 34.0
},
"niah_multikey_3": {
"string_match": 1.6
},
"niah_multiquery": {
"string_match": 58.9
},
"niah_multivalue": {
"string_match": 54.0
},
"niah_single_1": {
"string_match": 70.6
},
"niah_single_2": {
"string_match": 37.2
},
"niah_single_3": {
"string_match": 0.8
},
"qa_1": {
"string_match": 54.6
},
"qa_2": {
"string_match": 42.4
},
"vt": {
"string_match": 34.84
}
}

However, the KVzip's average metric is 93.99 in https://huggingface.co/spaces/nvidia/kvpress-leaderboard

[maxjeblick]

Thanks a lot for posting, we will inspect the differences.

[xhchen28]

Hello, sorry to bother you. I'd like to ask if there have been any updates—under the same setting, what results can you achieve?

[maxjeblick]

Hi! We will inspect the issue next week; it may take some time to pinpoint why results won't match. We'll update our findings in this issue.

[giulio98]

I observed the same issue, for example on LongBench benchmark, where I observed KVZip results considerably lower than Expected Attention.

[maxjeblick]

Hi, sorry for the late reply!

First of all, I checked kvzip's performance using kvpress==0.2.10 version; metrics are a lot better and align with the lb scores:

2025-11-06 22:36:52,629 - INFO - Inference completed.
2025-11-06 22:36:52,641 - INFO - Results saved to results/ruler__4096__Qwen--Qwen3-8B__kvzip__0.50/3/predictions.csv
2025-11-06 22:36:52,641 - INFO - Calculating metrics for dataset: ruler
2025-11-06 22:36:52,652 - INFO - Metrics saved to results/ruler__4096__Qwen--Qwen3-8B__kvzip__0.50/3/metrics.json
2025-11-06 22:36:52,653 - INFO - Average compression ratio: 0.50
2025-11-06 22:36:52,653 - INFO - Metrics:
{
  "cwe": {
    "string_match": 99.1
  },
  "fwe": {
    "string_match": 95.73
  },
  "niah_multikey_1": {
    "string_match": 100.0
  },
  "niah_multikey_2": {
    "string_match": 100.0
  },
  "niah_multikey_3": {
    "string_match": 100.0
  },
  "niah_multiquery": {
    "string_match": 99.95
  },
  "niah_multivalue": {
    "string_match": 99.95
  },
  "niah_single_1": {
    "string_match": 100.0
  },
  "niah_single_2": {
    "string_match": 100.0
  },
  "niah_single_3": {
    "string_match": 100.0
  },
  "qa_1": {
    "string_match": 80.0
  },
  "qa_2": {
    "string_match": 62.8
  },
  "vt": {
    "string_match": 100.0
  }
}

As for the degraded performance, we are still trying to find the regression.

We are currently focusing on #115. This PR was introduced because transformers library refactored their attention implementation significantly (see details and linked issues in the issue above).
Our current hypothesis is that kvzip's implementation is not fully compatible with new transformers version.

For now, I'd recommend using kvpress==0.2.10 when conducting experiments with kvzip.

cc @Janghyun1230 for visibilty.

[maxjeblick]

Seems I found the bug - it's a one liner in the end!

I'm currently running eval with qwen@50cr; will update with results. We will review the PR carefully (running more evals) and then merge. Thanks again for spotting this issue!

[xhchen28]

Seems I found the bug - it's a one liner in the end!

I'm currently running eval with qwen@50cr; will update with results. We will review the PR carefully (running more evals) and then merge. Thanks again for spotting this issue!

Thank you for your time and for looking into the issue. I appreciate your help and look forward to the resolution.

[maxjeblick]

Fixed by #152