Eval bug: Inference fails with certain inputs for Granite and Nemotron

[samteezy]

Name and Version

b7783

Operating systems

Linux

GGML backends

Vulkan

Hardware

Intel ARC Pro B50
AMD RX 7600XT

Models

unsloth/Nemotron-3-Nano-30B-A3B-UD-Q4_K_XL.gguf
unsloth/granite-4.0-h-small-UD-Q4_K_XL.gguf

Problem description & steps to reproduce

Pretty simple prompt - I'm copying and pasting the llama-server README.md file into a prompt and asking the model a simple question about it ("explain adaptive-p sampling"). This is happening in Open WebUI and in the llama.cpp web UI.

Nemotron fails immediately with this error message in Open WebUI:

------------
While executing FilterExpression at line 8, column 66 in source:
...~ json_key ~ '>' ~ (json_dict[json_key] | string) ~ '</' ~ json_key ~ '>' }}↵   ...
                                           ^
Error: Unknown (built-in) filter 'string' for type None

Granite starts giving me gibberish:

cision (I’m literally the same, but I’m not sure if you’re aware of the way to do it. I’m not sure if you’re aware of the way to do it, but you’ll have to download [I stopped it at this point]

Running this with unsloth/LFM2.5-1.2B-Instruct-UD-Q6_K_XL.gguf works just fine.

First Bad Commit

Not sure

Relevant log output

Pasted above. Not seeing any other relevant log info

[jeffbolznv]

Please check if this happens with --no-direct-io, and if it still does please try to bisect it.

[pwilkin]

Yes, there's something weird going on with the Nemotron Nano chat template support. As for Granite, might be just a template bug, we might have to add a string filter for None (@ngxson @CISC )

[pwilkin]

@jeffbolznv Nah, this is purely a template issue methinks.

[stephensrmmartin]

I'm experiencing this issue with unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:Q4_K_M as of today, and was not experiencing this just a few days ago. Same GGUF, same configuration as far as I can tell, but experiencing this error now.

@jeffbolznv It fails with both direct-io on and off.

Edit: I will bisect this now.

[stephensrmmartin]

The bad commit: c15395f

commit c15395f73c42805d9609a55749a1d4b5b2251379 (tag: b7756)
Author: Xuan-Son Nguyen <son@huggingface.co>
Date:   Fri Jan 16 11:22:06 2026 +0100

    common : implement new jinja template engine (#18462)
    
    * jinja vm
    
    * lexer
    
    * add vm types
    
    * demo
    
    * clean up
    
    * parser ok
    
    * binary_expression::execute
    
    * shadow naming
    
    * bin ops works!
    
    * fix map object
    
    * add string builtins
    
    * add more builtins
    
    * wip
    
    * use mk_val
    
    * eval with is_user_input
    
    * render gemma tmpl ok
    
    * track input string even after transformations
    
    * support binded functions
    
    * keyword arguments and slicing array
    
    * use shared_ptr for values
    
    * add mk_stmt
    
    * allow print source on exception
    
    * fix negate test
    
    * testing more templates
    
    * mostly works
    
    * add filter_statement
    
    * allow func to access ctx
    
    * add jinja-value.cpp
    
    * impl global_from_json
    
    * a lot of fixes
    
    * more tests
    
    * more fix, more tests
    
    * more fixes
    
    * rm workarounds
    
    * demo: type inferrence
    
    * add placeholder for tojson
    
    * improve function args handling
    
    * rm type inference
    
    * no more std::regex
    
    * trailing spaces
    
    * make testing more flexible
    
    * make output a bit cleaner
    
    * (wip) redirect minja calls
    
    * test: add --output
    
    * fix crash on macro kwargs
    
    * add minimal caps system
    
    * add some workarounds
    
    * rm caps_apply_workarounds
    
    * get rid of preprocessing
    
    * more fixes
    
    * fix test-chat-template
    
    * move test-chat-jinja into test-chat-template
    
    * rm test-chat-jinja from cmake
    
    * test-chat-template: use common
    
    * fix build
    
    * fix build (2)
    
    * rename vm --> interpreter
    
    * improve error reporting
    
    * correct lstrip behavior
    
    * add tojson
    
    * more fixes
    
    * disable tests for COMMON_CHAT_FORMAT_GENERIC
    
    * make sure tojson output correct order
    
    * add object.length
    
    * fully functional selectattr / rejectattr
    
    * improve error reporting
    
    * more builtins added, more fixes
    
    * create jinja rendering tests
    
    * fix testing.h path
    
    * adjust whitespace rules
    
    * more fixes
    
    * temporary disable test for ibm-granite
    
    * r/lstrip behavior matched with hf.js
    
    * minimax, glm4.5 ok
    
    * add append and pop
    
    * kimi-k2 ok
    
    * test-chat passed
    
    * fix lstrip_block
    
    * add more jinja tests
    
    * cast to unsigned char
    
    * allow dict key to be numeric
    
    * nemotron: rm windows newline
    
    * tests ok
    
    * fix test
    
    * rename interpreter --> runtime
    
    * fix build
    
    * add more checks
    
    * bring back generic format support
    
    * fix Apertus
    
    * [json.exception.out_of_range.403] key 'content' not found
    
    * rm generic test
    
    * refactor input marking
    
    * add docs
    
    * fix windows build
    
    * clarify error message
    
    * improved tests
    
    * split/rsplit with maxsplit
    
    * non-inverse maxsplit
    
    forgot to change after simplifying
    
    * implement separators for tojson and fix indent
    
    * i like to move it move it
    
    * rename null -- > none
    
    * token::eof
    
    * some nits + comments
    
    * add exception classes for lexer and parser
    
    * null -> none
    
    * rename global -> env
    
    * rm minja
    
    * update docs
    
    * docs: add input marking caveats
    
    * imlement missing jinja-tests functions
    
    * oops
    
    * support trim filter with args, remove bogus to_json reference
    
    * numerous argument fixes
    
    * updated tests
    
    * implement optional strip chars parameter
    
    * use new chars parameter
    
    * float filter also has default
    
    * always leave at least one decimal in float string
    
    * jinja : static analysis + header cleanup + minor fixes
    
    * add fuzz test
    
    * add string.cpp
    
    * fix chat_template_kwargs
    
    * nits
    
    * fix build
    
    * revert
    
    * unrevert
    
    sorry :)
    
    * add fuzz func_args, refactor to be safer
    
    * fix array.map()
    
    * loosen ensure_vals max count condition, add not impl for map(int)
    
    * hopefully fix windows
    
    * check if empty first
    
    * normalize newlines
    
    ---------
    
    Co-authored-by: Alde Rojas <hello@alde.dev>
    Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
    Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

That makes sense. Something is off with this commit, or the templates for these models are written using a known issue that was addressed here?

[CISC]

Yes, there's something weird going on with the Nemotron Nano chat template support. As for Granite, might be just a template bug, we might have to add a string filter for None (@ngxson @CISC )

Yes, that is indeed missing and works with jinja, feel free to PR.

[CISC]

That makes sense. Something is off with this commit, or the templates for these models are written using a known issue that was addressed here?

As already mentioned by @pwilkin the none type is missing the string filter, it's an easy fix. :)

[samteezy]

Thanks folks. I'm traveling for work and it's been difficult for me to get time to bisect it myself.