mtmd : chat : Fix extra \n between text and media marker by tdakhran · Pull Request #19595 · ggml-org/llama.cpp

tdakhran · 2026-02-13T15:21:12Z

Thanks to @tugot17 for detecting and reporting the issue.

For vision models (e.g. LFM2.5-VL-1.6B and Qwen/Qwen3-VL-4B-Instruct) llama-mtmd-cli produces identical output to HF implementation.

However llama-server doesn't. I traced it down to extra newline inserted after <__media__>.

This happens in to_json_oaicompat, that treats media markers as text and joins all parts with \n separator.

PR introduces new type media_marker and uses it for media markers. Extra logic is added to prevent insertion of newlines before and after media markers.

With this change number of input tokens is identical to HF implementation and as a result the output is also identical.

I explored other ways to address the issue

remove completely \n between text parts in to_json_oaicompat
merge text messages in server-common.cpp before sending them to to_json_oaicompat

Please propose alternative ways of fixing this issue.

ngxson · 2026-02-13T15:34:15Z

+                if (part.type != "text" && part.type != "media_marker") {
                    LOG_WRN("Ignoring content part type: %s\n", part.type.c_str());
                    continue;
                }
-                if (!text.empty()) {
+                if (part.type != "media_marker" && !last_was_media_marker && !text.empty()) {


for clarity, this should be refactored info:

if (part.type == "text") { ... } else if (part.type == "media_marker") { ... } else { LOG_WRN("Ignoring content part type: %s\n", part.type.c_str()); }

I'll wait CI run to complete in case some tests fail and then refactor.

@ngxson , shall I duplicate the body then in each of them? I'm struggling to understand how to split it because it's an && condition.

the body is small, it's fine to duplicate it. we just need to be explicit about what are possible values of type

refactored in b642c33

@tugot17

Thanks to @tugot17 for detecting and reporting the issue. For vision models (e.g. LFM2.5-VL-1.6B and Qwen/Qwen3-VL-4B-Instruct) `llama-mtmd-cli` produces identical output to HF implementation. However `llama-server` doesn't. I traced it down to extra newline inserted after `<__media__>`. This happens in `to_json_oaicompat`, that treats media markers as text and joins all parts with `\n` separator. PR introduces new type `media_marker` and uses it for media markers. Extra logic is added to prevent insertion of newlines before and after media markers. With this change number of input tokens is identical to HF implementation and as a result the output is also identical. I explored other ways to address the issue * remove completely `\n` between text parts in `to_json_oaicompat` * merge text messages in server-common.cpp before sending them to `to_json_oaicompat` Please propose alternative ways of fixing this issue.

tdakhran · 2026-02-17T13:48:20Z

Addressed existing comments. Please let me know if anything else is required to proceed with the merge.

pwilkin

Please just fix the whitespace stuff.

Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com>

tdakhran · 2026-02-17T15:26:23Z

I'm looking into server-windows CI failure, might be related.

tdakhran · 2026-02-17T15:36:25Z

Let's see now, updated legacy chat template handling.

tdakhran · 2026-02-17T21:34:32Z

CI passed, good to go.

tdakhran · 2026-02-19T08:28:36Z

@pwilkin, please let me know if anything is needed to proceed with the merge.

@tugot17

) * mtmd : chat : Fix extra \n between text and media marker Thanks to @tugot17 for detecting and reporting the issue. For vision models (e.g. LFM2.5-VL-1.6B and Qwen/Qwen3-VL-4B-Instruct) `llama-mtmd-cli` produces identical output to HF implementation. However `llama-server` doesn't. I traced it down to extra newline inserted after `<__media__>`. This happens in `to_json_oaicompat`, that treats media markers as text and joins all parts with `\n` separator. PR introduces new type `media_marker` and uses it for media markers. Extra logic is added to prevent insertion of newlines before and after media markers. With this change number of input tokens is identical to HF implementation and as a result the output is also identical. I explored other ways to address the issue * remove completely `\n` between text parts in `to_json_oaicompat` * merge text messages in server-common.cpp before sending them to `to_json_oaicompat` Please propose alternative ways of fixing this issue. * Refactor to use explicite per type ifs * Update common/chat.cpp Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com> * Update common_chat_templates_apply_legacy --------- Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com>

@tugot17

) * mtmd : chat : Fix extra \n between text and media marker Thanks to @tugot17 for detecting and reporting the issue. For vision models (e.g. LFM2.5-VL-1.6B and Qwen/Qwen3-VL-4B-Instruct) `llama-mtmd-cli` produces identical output to HF implementation. However `llama-server` doesn't. I traced it down to extra newline inserted after `<__media__>`. This happens in `to_json_oaicompat`, that treats media markers as text and joins all parts with `\n` separator. PR introduces new type `media_marker` and uses it for media markers. Extra logic is added to prevent insertion of newlines before and after media markers. With this change number of input tokens is identical to HF implementation and as a result the output is also identical. I explored other ways to address the issue * remove completely `\n` between text parts in `to_json_oaicompat` * merge text messages in server-common.cpp before sending them to `to_json_oaicompat` Please propose alternative ways of fixing this issue. * Refactor to use explicite per type ifs * Update common/chat.cpp Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com> * Update common_chat_templates_apply_legacy --------- Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com>

@tugot17

) * mtmd : chat : Fix extra \n between text and media marker Thanks to @tugot17 for detecting and reporting the issue. For vision models (e.g. LFM2.5-VL-1.6B and Qwen/Qwen3-VL-4B-Instruct) `llama-mtmd-cli` produces identical output to HF implementation. However `llama-server` doesn't. I traced it down to extra newline inserted after `<__media__>`. This happens in `to_json_oaicompat`, that treats media markers as text and joins all parts with `\n` separator. PR introduces new type `media_marker` and uses it for media markers. Extra logic is added to prevent insertion of newlines before and after media markers. With this change number of input tokens is identical to HF implementation and as a result the output is also identical. I explored other ways to address the issue * remove completely `\n` between text parts in `to_json_oaicompat` * merge text messages in server-common.cpp before sending them to `to_json_oaicompat` Please propose alternative ways of fixing this issue. * Refactor to use explicite per type ifs * Update common/chat.cpp Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com> * Update common_chat_templates_apply_legacy --------- Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com>

@tugot17

) * mtmd : chat : Fix extra \n between text and media marker Thanks to @tugot17 for detecting and reporting the issue. For vision models (e.g. LFM2.5-VL-1.6B and Qwen/Qwen3-VL-4B-Instruct) `llama-mtmd-cli` produces identical output to HF implementation. However `llama-server` doesn't. I traced it down to extra newline inserted after `<__media__>`. This happens in `to_json_oaicompat`, that treats media markers as text and joins all parts with `\n` separator. PR introduces new type `media_marker` and uses it for media markers. Extra logic is added to prevent insertion of newlines before and after media markers. With this change number of input tokens is identical to HF implementation and as a result the output is also identical. I explored other ways to address the issue * remove completely `\n` between text parts in `to_json_oaicompat` * merge text messages in server-common.cpp before sending them to `to_json_oaicompat` Please propose alternative ways of fixing this issue. * Refactor to use explicite per type ifs * Update common/chat.cpp Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com> * Update common_chat_templates_apply_legacy --------- Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com>

@tugot17

… and new jinja template engine (ggml-org#1369) --------- Co-authored-by: Piotr Wilkin <piotr.wilkin@syndatis.com> common : add nemotron 3 parsing (ggml-org#18077) common : add parser for ministral/mistral large 3/devstral 2 (ggml-org#17713) common : default content to an empty string (ggml-org#18485) chat: make tool description and parameters optional per OpenAI spec (ggml-org#18478) Per the OpenAI API specification, both 'description' and 'parameters' fields in tool function definitions are optional. Previously, the parser would throw an exception if these fields were missing. Attempts to fix ggml-org#17667 common : implement new jinja template engine (ggml-org#18462) --------- Co-authored-by: Alde Rojas <hello@alde.dev> Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> jinja: correct member access rule (ggml-org#18905) jinja : fix lexing of float literals with sign (ggml-org#18901) jinja : add missing tojson filter for bool (ggml-org#18900) jinja : attribute support for join, map and sort (ggml-org#18883) jinja : fix object item order (and properly implement dictsort) (ggml-org#18904) tests : add test-jinja -py option for cross-checking (ggml-org#18906) Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> ci : run test-jinja -py on high perf [no ci] (ggml-org#18916) jinja : fix undefined keys and attributes and int/float as bool (ggml-org#18924) jinja: support none|string (ggml-org#18995) Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> jinja : implement mixed type object keys (ggml-org#18955) --------- Co-authored-by: Xuan Son Nguyen <son@huggingface.co> jinja : undefined should be treated as sequence/iterable (return string/array) by filters/tests (ggml-org#19147) `tojson` is not a supported `undefined` filter keep it DRY and fix some types jinja : do not pass empty tools and add some none filters (ggml-org#19176) jinja : add unordered_map include to value.h [no ci] (ggml-org#19205) jinja : add missing 'in' test to template engine (ggml-org#19004) (ggml-org#19239) The jinja template parser was missing the 'in' test from global_builtins(), causing templates using reject("in", ...), select("in", ...), or 'x is in(y)' to fail with "selectattr: unknown test 'in'". This broke tool-calling for Qwen3-Coder and any other model whose chat template uses the 'in' test. Added test_is_in supporting array, string, and object containment checks, mirroring the existing 'in' operator logic in runtime.cpp. Includes test cases for all three containment types plus reject/select filter usage. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Sid Mohan <sidmohan0@users.noreply.github.com> Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> Co-authored-by: Xuan Son Nguyen <son@huggingface.co> Add Jinja support for "indent" string filter (ggml-org#19529) Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> add vendor refactor chat server : support preserving reasoning_content in assistant message (ggml-org#18994) chat : fix translategemma crash on common_chat_format_example (ggml-org#19019) chat: fix language input for translategemma (ggml-org#19052) Co-authored-by: Aldehir Rojas <hello@alde.dev> --------- Co-authored-by: Aldehir Rojas <hello@alde.dev> chat: fix case where template accepts type content only (ggml-org#19419) mtmd : chat : Fix extra \n between text and media marker (ggml-org#19595) Thanks to @tugot17 for detecting and reporting the issue. For vision models (e.g. LFM2.5-VL-1.6B and Qwen/Qwen3-VL-4B-Instruct) `llama-mtmd-cli` produces identical output to HF implementation. However `llama-server` doesn't. I traced it down to extra newline inserted after `<__media__>`. This happens in `to_json_oaicompat`, that treats media markers as text and joins all parts with `\n` separator. PR introduces new type `media_marker` and uses it for media markers. Extra logic is added to prevent insertion of newlines before and after media markers. With this change number of input tokens is identical to HF implementation and as a result the output is also identical. I explored other ways to address the issue * remove completely `\n` between text parts in `to_json_oaicompat` * merge text messages in server-common.cpp before sending them to `to_json_oaicompat` Please propose alternative ways of fixing this issue. Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com> --------- Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com> common : merge qwen3-coder and nemotron nano 3 parsers (ggml-org#19765) common : fix improper trimming in XML parser on complete message (ggml-org#19805) Co-authored-by: Jules LEIDELINGER <11395311+julio75012@users.noreply.github.com> jinja: correct stats for tojson and string filters (ggml-org#19785) jinja : correct default size for string slices (ggml-org#19913) common : handle unicode during partial json parsing (ggml-org#16526) common : fix json schema with '\' in literals (ggml-org#17307) add back qwen_coder_xml and mirothinker Co-authored-by: Aldehir Rojas <hello@alde.dev>

@tugot17

) * mtmd : chat : Fix extra \n between text and media marker Thanks to @tugot17 for detecting and reporting the issue. For vision models (e.g. LFM2.5-VL-1.6B and Qwen/Qwen3-VL-4B-Instruct) `llama-mtmd-cli` produces identical output to HF implementation. However `llama-server` doesn't. I traced it down to extra newline inserted after `<__media__>`. This happens in `to_json_oaicompat`, that treats media markers as text and joins all parts with `\n` separator. PR introduces new type `media_marker` and uses it for media markers. Extra logic is added to prevent insertion of newlines before and after media markers. With this change number of input tokens is identical to HF implementation and as a result the output is also identical. I explored other ways to address the issue * remove completely `\n` between text parts in `to_json_oaicompat` * merge text messages in server-common.cpp before sending them to `to_json_oaicompat` Please propose alternative ways of fixing this issue. * Refactor to use explicite per type ifs * Update common/chat.cpp Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com> * Update common_chat_templates_apply_legacy --------- Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com>

tdakhran requested review from ggerganov, ngxson and pwilkin as code owners February 13, 2026 15:21

github-actions Bot added examples server labels Feb 13, 2026

ngxson reviewed Feb 13, 2026

View reviewed changes

Comment thread tools/server/server-common.cpp

ngxson reviewed Feb 13, 2026

View reviewed changes

tdakhran added 2 commits February 17, 2026 14:43

Refactor to use explicite per type ifs

b642c33

tdakhran force-pushed the tarek/fix/extra-newline-with-media-markers branch from 2e17f6a to b642c33 Compare February 17, 2026 13:44

tdakhran requested a review from ngxson February 17, 2026 13:47

pwilkin approved these changes Feb 17, 2026

View reviewed changes

Comment thread common/chat.cpp Outdated

Comment thread common/chat.cpp

Update common/chat.cpp

de2ae13

Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com>

Update common_chat_templates_apply_legacy

8be2896

ngxson approved these changes Feb 19, 2026

View reviewed changes

ngxson merged commit c589799 into ggml-org:master Feb 19, 2026
78 checks passed

tdakhran deleted the tarek/fix/extra-newline-with-media-markers branch February 19, 2026 11:19

Conversation

tdakhran commented Feb 13, 2026

Uh oh!

Uh oh!

ngxson Feb 13, 2026

Choose a reason for hiding this comment

Uh oh!

tdakhran Feb 13, 2026

Choose a reason for hiding this comment

Uh oh!

tdakhran Feb 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ngxson Feb 13, 2026

Choose a reason for hiding this comment

Uh oh!

tdakhran Feb 17, 2026

Choose a reason for hiding this comment

Uh oh!

tdakhran commented Feb 17, 2026

Uh oh!

pwilkin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

tdakhran commented Feb 17, 2026

Uh oh!

tdakhran commented Feb 17, 2026

Uh oh!

tdakhran commented Feb 17, 2026

Uh oh!

tdakhran commented Feb 19, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

tdakhran Feb 13, 2026 •

edited

Loading