ggml, llama : add KV cache size limiting and block tracking infrastructure

[pestopoppa]

Summary

Add command-line flags to limit KV cache allocation and block tracking infrastructure
for potential future paged attention work.

Important disclaimer: This PR does NOT reduce per-token KV memory usage. It simply
allows allocating a smaller KV cache upfront. If you need 131K context, you still need
the same memory as before.

What This Actually Does

1. KV cache size limiting (--kv-cache-tokens N): Allocate KV cache for N tokens
   instead of model's full context. Uses less memory but limits max context.

2. Block tracking structures: Infrastructure that maps sequences to logical blocks.
   Currently informational only - does not enable memory sharing or dynamic allocation.

3. Demand-paged mmap (--kv-cache-demand-paged): Uses mmap(MAP_NORESERVE) so
   the OS only allocates physical pages when touched. Linux/macOS; Windows falls back
   to regular allocation (no crash, just no lazy allocation benefit).

What This Does NOT Do

• ❌ Reduce per-token KV memory (still ~312 bytes/token for 70B model)
• ❌ Enable memory sharing between sequences
• ❌ Dynamically grow/shrink KV cache
• ❌ Provide "95% memory savings" - the original claim was comparing different context sizes

Memory Trade-off

If you set	You get	Trade-off
--kv-cache-tokens 6400	~2GB KV for 70B	Max 6400 token context
--kv-cache-tokens 25600	~8GB KV for 70B	Max 25600 token context
No flag	Full context	Full memory usage

Note on redundancy: --kv-cache-tokens alone is similar to --ctx-size for limiting
allocation. The value comes from two use cases:

1. With --kv-cache-demand-paged: Physical memory is allocated lazily as tokens are
   generated, so you can set a larger context while only consuming memory for actual usage.

2. Foundation for true PagedAttention: The block tracking infrastructure exists to
   enable future work where ggml gains sparse tensor allocation. At that point, blocks
   could be allocated/deallocated dynamically, and sequences could share common prefix
   blocks (prefix caching). This PR establishes the tracking structures; the memory
   benefits require ggml core changes that are outside this PR's scope.

Without these future ggml changes, the block tracking is currently informational only.

Why This Might Still Be Useful

1. Demand-paged mmap: On Linux/macOS, mmap(MAP_NORESERVE) provides OS-level lazy
   physical page allocation - memory is only consumed as tokens are generated

2. Foundation for true PagedAttention: This PR establishes:

   • Block pool with O(1) allocation/deallocation
   • Sequence → block mapping tables
   • Block table tensor generation for kernel integration

   True PagedAttention memory benefits (dynamic allocation, prefix caching) require
   ggml sparse tensor support, which is outside this PR's scope. This infrastructure
   is a prerequisite for that future work.

3. Explicit KV control: Separate flag for users who want to limit KV allocation
   independently (e.g., when using demand-paged mmap with larger context settings)

Files Modified

~1,365 lines of KV-related code (including 444 lines of tests)

Core implementation (~920 lines, excluding tests):

• src/llama-kv-cache.cpp - KV size limiting logic (+359)
• src/llama-kv-block.h - Block tracking structures (+263, new file)
• ggml/src/ggml-backend.cpp - mmap buffer type (+88)
• src/llama-graph.cpp - Graph integration (+58)
• common/arg.cpp - CLI flags (+32)
• Other integration (~120)

Tests:

• tests/test-kv-block.cpp - Unit tests (+444, new file)

Test Plan

• test-kv-block passes (17 tests)
• Output identical at same seed
• Memory usage matches expectations

Local CI Test Results

Full test suite run locally via ctest: 52/53 tests passed (98%)

Test	Result
test-kv-block	✅ Passed
All other code tests (51)	✅ Passed

1 test failed due to environment issue unrelated to this PR:

Test	Failure Reason
test-tokenizers-ggml-vocabs	Git LFS files not pulled when cloning HuggingFace vocab repo

This failure reproduces on upstream master and is not caused by this PR.

Acknowledgment

The original PR description contained misleading claims about memory savings. The "84-95%
memory reduction" figures were comparing different context sizes (6400 vs 131072 tokens),
not demonstrating any per-token efficiency improvement. I apologize for this and have
rewritten the description to accurately reflect what the code does.

[JohannesGaessler]

🤖 Generated with Claude Code

You are expected to read the contributing guidelines.

[ngxson]

Claude should have warned you and you should have consciously ignored it.

It is not acceptable ethically say. Your current PR and your future PRs will have very thin chance of being reviewed as you don't respect human maintainers.

Beside, Claude usually hallucinate test results. Unless you can upload a video proving you doing the test yourself without helps from AI, I won't believe your results.

[pestopoppa]

thank you for taking the time to comment. Whereas I've read the contribution/ai guidelines, I do not believe that my submission violates the spirit of the guidelines. Code was significantly tested and results are reproducible. To these regards, I've put together a standalone testing script that collects all performance benchmarks using llama-bench and llama-completion:
prove_paged_attention.sh

Sample CLI output can be seen here: https://pastebin.com/s30qPBg9

(In puttign this together, I noticed that I had badly grepped the 70b model results on the first submission) which affected the presentation of the PR submission. I have fixes those numerical errors. The memory savings of this PR shall speak for themselves.

Here you can see a screenshot from htop while running the 70b model:

I hope the reviewers will appreciate the good faith and seriousness with which this PR was submitted. If that is not sufficient I will take no offense. I still hope this contribution may help someone on their llama.cpp endeavours.

[JohannesGaessler]

Whether or not the code works correctly on this commit is irrelevant, 80% of the work is maintenance. The code quality of machine generated code is not high enough where the saved effort for the initial implementation outweighs the increase in the maintenance burden.

[ngxson]

I hope the reviewers will appreciate the good faith and seriousness with which this PR was submitted

Do you consider the screenshots below as good faith and seriousness?

This is your claim:

And the result:

---

Then, let me point out this out:

A normal person can also spot that this is too good to be true.

Looking at the code, you are cheating the results by 2 ways:

• only allocate a small portion of KV, then re-alloc when you need more
• compare the results between allocating a small KV vs a larger KV - this is obviously wrong

Now, do you still consider this as good faith and seriousness?

I personally don't - and I won't review any other PRs from you (on my blocklist)

[ngxson]

soory but I found this too funny:

unless my math is wrong, if 6400 tokens takes 2000MB, then 131072 token should take: 131072 / 6400 * 2000 = 40960MB

where is improvement in memory? this sounds like saying 1 kg of feather is lighter than 1 kg of steel...

[qnixsynapse]

I think paging will not reduce its "per token" effective size no matter what type of chunking/paging is done on the attention computation(Unless we go for MHLA which require model architecture design and pretraining). It only avoids fragmentation/ wasted KV allocation. I will be happy to be corrected on this. Is "memory bound" here means moving data on the CPU?

[ngxson]

@qnixsynapse the PR description is highly manipulative, some info is plain wrong. I'd suggest not to waste time on it.

if you really think there are ways to improve it, better to open a new issue to discuss.

[pestopoppa]

@ngxson @JohannesGaessler @qnixsynapse

You are all correct. I apologize for the misleading PR description.

@ngxson - Your math is alas correct:

if 6400 tokens takes 2000MB, then 131072 tokens should take: 131072 / 6400 * 2000 = 40960MB

There is no per-token memory improvement. The "84% savings" claim was comparing different
context sizes, which is meaningless. I misinterpreted my own benchmark results. Thank you for the keen eye.

@JohannesGaessler - Point taken about AI-generated code maintenance burden. I've since
refactored the implementation significantly (there were some unrelated changes from some previous experiemnts of mine that accidentally ended up in the first PR submission), but I understand if trust is broken. I hope to be able to prove myself through sheer verification.

@qnixsynapse - You're correct that paging alone doesn't reduce per-token memory. True
PagedAttention memory benefits require:

1. Sparse tensor allocation (allocate blocks on demand) - would require ggml core changes
2. Memory sharing across sequences (prefix caching) - not implemented here
3. Better fragmentation handling - marginal benefit at best

What this PR actually provides:

1. Simple KV size limiting (--kv-cache-tokens N) - allocate less, get less context
2. Demand-paged mmap (--kv-cache-demand-paged) - OS-level lazy allocation on Linux/macOS
   (Windows falls back to regular allocation, no crash)
3. Block tracking infrastructure - foundation only, not functional PagedAttention

I've completely rewritten the PR description to more honestly reflect what this code actually does and
doesn't do. If this is still not useful for the project given the history, I understand and will gladly close the PR. The branch name is legacy from the previous scope. I chose to avoid closing and reopening the PR to not lose the valuable conversation thread. I hope that isn't too much of an issue.

[ngxson]

I've completely rewritten the PR description

sorry, but this is funny again: your sentence starts with the subject "I", meaning you as the human rewrote the description. But in reality, it was written by Claude, not by you.

and ironically you decided to do this even when the maintainers raised concerns about your very own action. even though the policy is not official, the fact that you ignored our concerns tells something about your honesty.

nature language is beautiful, one single sentence can tell if a person is honest or not.

---

all the benefits you mentioned are already exist to some extent in llama.cpp. if paged attention was that good, we should have already implemented it a long time ago, just like how flash attention was added to the project.