PEG parser for LFM2

[pwilkin]

Dedicated parser for LFM2, this is the only model using Python-style tools for now, so I won't integrate it into the autoparser yet until more templates in this style show up.

Fixes #20245

[eapache]

🤔 surprisingly @pwilkin, this hasn't actually fixed the issue with LFM2.5-Instruct. Do I need to add a cli flag to force the use of the PEG parser instead of the autoparser?

edit: testing against build: 8262 (23fbfcb1a)

Not sure if this is irrelevant or if it indicates that the code in this PR is not being invoked properly, but I still get the following on boot:

[48973] Using differential autoparser
[48973] === Starting differential analysis ===
[48973] Phase 1: Reasoning analysis
[48973] Phase 2: Content analysis
[48973] 
[48973] --- Reasoning & Content Structure ---
[48973] reasoning_mode: NONE
[48973] reasoning_start: ''
[48973] reasoning_end: ''
[48973] content_mode: PLAIN
[48973] content_start: ''
[48973] content_end: ''
[48973] 
[48973] --- Tool Call Structure ---
[48973] tool_mode: NONE
[48973] supports_tools: true
[48973] supports_parallel_calls: false
[48973] tool_section_start: ''
[48973] tool_section_end: ''
[48973] per_call_start: ''
[48973] per_call_end: ''
[48973] func_name_prefix: ''
[48973] func_name_suffix: ''
[48973] func_close: ''
[48973] python_dict_format: false
[48973] arg_name_prefix: ''
[48973] arg_name_suffix: ''
[48973] arg_value_prefix: ''
[48973] arg_value_suffix: ''
[48973] name_field: 'name'
[48973] args_field: 'arguments'
[48973] id_field: ''
[48973] gen_id_field: ''
[48973] parameter_order: ''

And the following verbose logs from the middle of the request, right as the tool call is completed:

Logs

[48973] res          send: sending result for task id = 0
[48973] res          send: task id = 0 pushed to result queue
[48973] slot process_toke: id  3 | task 0 | n_decoded = 35, n_remaining = -1, next token:  6681 '")'
[48973] srv  update_slots: run slots completed
[48973] que    start_loop: waiting for new tasks
[48973] que    start_loop: processing new tasks
[48973] que    start_loop: processing task, id = 38
[48973] que    start_loop: update slots
[48973] srv  update_slots: posting NEXT_RESPONSE
[48973] que          post: new task, id = 39, front = 0
[48973] srv  update_chat_: Parsing chat message: [brave_news_search(query="Ottawa Senators next game schedule", country="CA", search_lang="en", freshness="pd")
[48973] Parsing PEG input with format peg-native: [brave_news_search(query="Ottawa Senators next game schedule", country="CA", search_lang="en", freshness="pd")
[48973] slot update_slots: id  3 | task 0 | slot decode token, n_ctx = 128000, n_tokens = 6185, truncated = 0
[48973] srv  update_slots: decoding batch, n_tokens = 1
[48973] set_adapters_lora: adapters = (nil)
[48973] adapters_lora_are_same: adapters = (nil)
[48973] set_embeddings: value = 0
[48973] srv    operator(): http: streamed chunk: data: {"choices":[{"finish_reason":null,"index":0,"delta":{"content":"\")"}}],"created":1773076140,"id":"chatcmpl-FJOABg0Exz1kxtxeUn80RrxXqFEGLFrU","model":"LFM2.5-Instruct","system_fingerprint":"b8262-23fbfcb1a","object":"chat.completion.chunk","timings":{"cache_n":0,"prompt_n":6150,"prompt_ms":573.299,"prompt_per_token_ms":0.09321934959349593,"prompt_per_second":10727.386581870891,"predicted_n":35,"predicted_ms":197.35,"predicted_per_token_ms":5.638571428571429,"predicted_per_second":177.34988598935902}}
[48973] 
[48973] 
srv    operator(): http: streamed chunk: data: {"choices":[{"finish_reason":null,"index":0,"delta":{"content":"\")"}}],"created":1773076140,"id":"chatcmpl-FJOABg0Exz1kxtxeUn80RrxXqFEGLFrU","model":"LFM2.5-Instruct","system_fingerprint":"b8262-23fbfcb1a","object":"chat.completion.chunk","timings":{"cache_n":0,"prompt_n":6150,"prompt_ms":573.299,"prompt_per_token_ms":0.09321934959349593,"prompt_per_second":10727.386581870891,"predicted_n":35,"predicted_ms":197.35,"predicted_per_token_ms":5.638571428571429,"predicted_per_second":177.34988598935902}}


[48973] res          send: sending result for task id = 0
[48973] res          send: task id = 0 pushed to result queue
[48973] slot process_toke: id  3 | task 0 | n_decoded = 36, n_remaining = -1, next token:   570 ']'
[48973] srv  update_slots: run slots completed
[48973] que    start_loop: waiting for new tasks
[48973] que    start_loop: processing new tasks
[48973] que    start_loop: processing task, id = 39
[48973] que    start_loop: update slots
[48973] srv  update_slots: posting NEXT_RESPONSE
[48973] que          post: new task, id = 40, front = 0
[48973] srv  update_chat_: Parsing chat message: [brave_news_search(query="Ottawa Senators next game schedule", country="CA", search_lang="en", freshness="pd")]
[48973] Parsing PEG input with format peg-native: [brave_news_search(query="Ottawa Senators next game schedule", country="CA", search_lang="en", freshness="pd")]
[48973] slot update_slots: id  3 | task 0 | slot decode token, n_ctx = 128000, n_tokens = 6186, truncated = 0
[48973] srv  update_slots: decoding batch, n_tokens = 1
[48973] set_adapters_lora: adapters = (nil)
[48973] adapters_lora_are_same: adapters = (nil)
[48973] set_embeddings: value = 0
[48973] srv    operator(): http: streamed chunk: data: {"choices":[{"finish_reason":null,"index":0,"delta":{"content":"]"}}],"created":1773076140,"id":"chatcmpl-FJOABg0Exz1kxtxeUn80RrxXqFEGLFrU","model":"LFM2.5-Instruct","system_fingerprint":"b8262-23fbfcb1a","object":"chat.completion.chunk","timings":{"cache_n":0,"prompt_n":6150,"prompt_ms":573.299,"prompt_per_token_ms":0.09321934959349593,"prompt_per_second":10727.386581870891,"predicted_n":36,"predicted_ms":203.499,"predicted_per_token_ms":5.65275,"predicted_per_second":176.90504621644334}}
[48973] 
[48973] 
srv    operator(): http: streamed chunk: data: {"choices":[{"finish_reason":null,"index":0,"delta":{"content":"]"}}],"created":1773076140,"id":"chatcmpl-FJOABg0Exz1kxtxeUn80RrxXqFEGLFrU","model":"LFM2.5-Instruct","system_fingerprint":"b8262-23fbfcb1a","object":"chat.completion.chunk","timings":{"cache_n":0,"prompt_n":6150,"prompt_ms":573.299,"prompt_per_token_ms":0.09321934959349593,"prompt_per_second":10727.386581870891,"predicted_n":36,"predicted_ms":203.499,"predicted_per_token_ms":5.65275,"predicted_per_second":176.90504621644334}}


[48973] res          send: sending result for task id = 0
[48973] res          send: task id = 0 pushed to result queue
[48973] slot process_toke: id  3 | task 0 | n_decoded = 37, n_remaining = -1, next token:    11 ''
[48973] srv  update_slots: run slots completed
[48973] que    start_loop: waiting for new tasks
[48973] que    start_loop: processing new tasks
[48973] que    start_loop: processing task, id = 40
[48973] que    start_loop: update slots
[48973] srv  update_slots: posting NEXT_RESPONSE
[48973] que          post: new task, id = 41, front = 0
[48973] srv  update_chat_: Parsing chat message: [brave_news_search(query="Ottawa Senators next game schedule", country="CA", search_lang="en", freshness="pd")]
[48973] Parsing PEG input with format peg-native: [brave_news_search(query="Ottawa Senators next game schedule", country="CA", search_lang="en", freshness="pd")]
[48973] slot update_slots: id  3 | task 0 | slot decode token, n_ctx = 128000, n_tokens = 6187, truncated = 0
[48973] srv  update_slots: decoding batch, n_tokens = 1
[48973] set_adapters_lora: adapters = (nil)
[48973] adapters_lora_are_same: adapters = (nil)
[48973] set_embeddings: value = 0
[48973] res          send: sending result for task id = 0
[48973] res          send: task id = 0 pushed to result queue
[48973] slot process_toke: id  3 | task 0 | n_decoded = 38, n_remaining = -1, next token:   550 'I'
[48973] srv  update_slots: run slots completed
[48973] que    start_loop: waiting for new tasks
[48973] que    start_loop: processing new tasks
[48973] que    start_loop: processing task, id = 41
[48973] que    start_loop: update slots
[48973] srv  update_slots: posting NEXT_RESPONSE
[48973] que          post: new task, id = 42, front = 0
[48973] srv  update_chat_: Parsing chat message: [brave_news_search(query="Ottawa Senators next game schedule", country="CA", search_lang="en", freshness="pd")]I
[48973] Parsing PEG input with format peg-native: [brave_news_search(query="Ottawa Senators next game schedule", country="CA", search_lang="en", freshness="pd")]I
[48973] slot update_slots: id  3 | task 0 | slot decode token, n_ctx = 128000, n_tokens = 6188, truncated = 0
[48973] srv  update_slots: decoding batch, n_tokens = 1
[48973] set_adapters_lora: adapters = (nil)
[48973] adapters_lora_are_same: adapters = (nil)
[48973] set_embeddings: value = 0
[48973] srv    operator(): http: streamed chunk: data: {"choices":[{"finish_reason":null,"index":0,"delta":{"content":"I"}}],"created":1773076140,"id":"chatcmpl-FJOABg0Exz1kxtxeUn80RrxXqFEGLFrU","model":"LFM2.5-Instruct","system_fingerprint":"b8262-23fbfcb1a","object":"chat.completion.chunk","timings":{"cache_n":0,"prompt_n":6150,"prompt_ms":573.299,"prompt_per_token_ms":0.09321934959349593,"prompt_per_second":10727.386581870891,"predicted_n":38,"predicted_ms":214.448,"predicted_per_token_ms":5.643368421052632,"predicted_per_second":177.19913452212188}}
[48973] 
[48973] 
srv    operator(): http: streamed chunk: data: {"choices":[{"finish_reason":null,"index":0,"delta":{"content":"I"}}],"created":1773076140,"id":"chatcmpl-FJOABg0Exz1kxtxeUn80RrxXqFEGLFrU","model":"LFM2.5-Instruct","system_fingerprint":"b8262-23fbfcb1a","object":"chat.completion.chunk","timings":{"cache_n":0,"prompt_n":6150,"prompt_ms":573.299,"prompt_per_token_ms":0.09321934959349593,"prompt_per_second":10727.386581870891,"predicted_n":38,"predicted_ms":214.448,"predicted_per_token_ms":5.643368421052632,"predicted_per_second":177.19913452212188}}


[48973] res          send: sending result for task id = 0
[48973] res          send: task id = 0 pushed to result queue
[48973] slot process_toke: id  3 | task 0 | n_decoded = 39, n_remaining = -1, next token:  6217 ''m'
[48973] srv  update_slots: run slots completed
[48973] que    start_loop: waiting for new tasks
[48973] que    start_loop: processing new tasks
[48973] que    start_loop: processing task, id = 42
[48973] que    start_loop: update slots
[48973] srv  update_slots: posting NEXT_RESPONSE
[48973] que          post: new task, id = 43, front = 0
[48973] slot update_slots: id  3 | task 0 | slot decode token, n_ctx = 128000, n_tokens = 6189, truncated = 0
[48973] srv  update_slots: decoding batch, n_tokens = 1
[48973] set_adapters_lora: adapters = (nil)
[48973] adapters_lora_are_same: adapters = (nil)
[48973] set_embeddings: value = 0
[48973] srv  update_chat_: Parsing chat message: [brave_news_search(query="Ottawa Senators next game schedule", country="CA", search_lang="en", freshness="pd")]I'm
[48973] Parsing PEG input with format peg-native: [brave_news_search(query="Ottawa Senators next game schedule", country="CA", search_lang="en", freshness="pd")]I'm
[48973] srv    operator(): http: streamed chunk: data: {"choices":[{"finish_reason":null,"index":0,"delta":{"content":"'m"}}],"created":1773076141,"id":"chatcmpl-FJOABg0Exz1kxtxeUn80RrxXqFEGLFrU","model":"LFM2.5-Instruct","system_fingerprint":"b8262-23fbfcb1a","object":"chat.completion.chunk","timings":{"cache_n":0,"prompt_n":6150,"prompt_ms":573.299,"prompt_per_token_ms":0.09321934959349593,"prompt_per_second":10727.386581870891,"predicted_n":39,"predicted_ms":220.459,"predicted_per_token_ms":5.652794871794872,"predicted_per_second":176.90364194702872}}
[48973] 
[48973] 
srv    operator(): http: streamed chunk: data: {"choices":[{"finish_reason":null,"index":0,"delta":{"content":"'m"}}],"created":1773076141,"id":"chatcmpl-FJOABg0Exz1kxtxeUn80RrxXqFEGLFrU","model":"LFM2.5-Instruct","system_fingerprint":"b8262-23fbfcb1a","object":"chat.completion.chunk","timings":{"cache_n":0,"prompt_n":6150,"prompt_ms":573.299,"prompt_per_token_ms":0.09321934959349593,"prompt_per_second":10727.386581870891,"predicted_n":39,"predicted_ms":220.459,"predicted_per_token_ms":5.652794871794872,"predicted_per_second":176.90364194702872}}

[aldehir]

I think it's because the template doesn't include the markers this code uses to detect if it's LFM-2.5. In fact, I'm not seeing much that can be used as a discriminator.

[eapache]

It was working properly pre-auto parser; what was used to discriminate in that code?

[aldehir]

Looks like it's the same tags.

    // LFM2 (w/ tools)
    if (src.find("List of tools: <|tool_list_start|>[") != std::string::npos &&
        src.find("]<|tool_list_end|>") != std::string::npos) {
        return common_chat_params_init_lfm2(tmpl, params);
    }

But LFM2.5 doesn't appear to use these tags, which means it must have been handled by the previous generic parser.

[ykhrustalev]

that change broke the lfm2 tool calling