Skip to content

main: use jinja chat template system prompt by default#12118

Merged
ngxson merged 5 commits intoggml-org:masterfrom
CISC:jinja-system-prompt
Mar 2, 2025
Merged

main: use jinja chat template system prompt by default#12118
ngxson merged 5 commits intoggml-org:masterfrom
CISC:jinja-system-prompt

Conversation

@CISC
Copy link
Copy Markdown
Member

@CISC CISC commented Feb 28, 2025

Using the new --jinja option would override the builtin system prompt with You are a helpful assistant if in conversation mode and/or no prompt was defined.

Fixed this behaviour so that -p option now either defines system prompt (in conversation mode) or user prompt, and does not fill in a default so that the chat template's default is used instead.

Adds some extra safety checks to be able to process an interactive user prompt with no defined system prompt.

@ochafik PTAL


No system prompt:

./build/bin/llama-cli -m phi-4-IQ2_M.gguf

 - Not using system message. To change it, set a different value via -sys PROMPT

> hi
Hello! How can I assist you today?

Incorrect use -p

./build/bin/llama-cli -m phi-4-IQ2_M.gguf -p test

*** User-specified prompt in conversation mode will be ignored, did you mean to set --system-prompt (-sys) instead?

 - Not using system message. To change it, set a different value via -sys PROMPT
...

Correctly using system prompt:

./build/bin/llama-cli -m phi-4-IQ2_M.gguf -sys "Your name is Sydney"

systemYour name is Sydney


> what is your name
My name is Sydney. How can I assist you today?

@ngxson
Copy link
Copy Markdown
Contributor

ngxson commented Mar 1, 2025

now either defines system prompt (in conversation mode) or user prompt

From UX perspective, this is very confusing for end-user. I would suggest that we either:

  • Simply remove default system prompt if jinja is set
  • Or, add a dedicated argument for system prompt, for example -sp, --system-prompt

Comment thread examples/main/main.cpp
@CISC
Copy link
Copy Markdown
Member Author

CISC commented Mar 1, 2025

now either defines system prompt (in conversation mode) or user prompt
From UX perspective, this is very confusing for end-user. I would suggest that we either:

Perhaps, but it is already doing this, just wrongly. :)

* Or, add a dedicated argument for system prompt, for example `-sp, --system-prompt`

Makes sense, but maybe in a different PR?

@ngxson
Copy link
Copy Markdown
Contributor

ngxson commented Mar 1, 2025

Makes sense, but maybe in a different PR?

Yes and it would be better if you we can merge that PR (adding -sp, --system-prompt) before merging this one.

@CISC
Copy link
Copy Markdown
Member Author

CISC commented Mar 1, 2025

Makes sense, but maybe in a different PR?
Yes and it would be better if you we can merge that PR (adding -sp, --system-prompt) before merging this one.

Sure, I'll take a look.

@CISC
Copy link
Copy Markdown
Member Author

CISC commented Mar 1, 2025

@ngxson See #12131 (-sp was already taken).

@CISC CISC changed the title Use jinja chat template system prompt by default main: use jinja chat template system prompt by default Mar 1, 2025
@CISC CISC requested a review from ngxson March 1, 2025 13:35
@CISC
Copy link
Copy Markdown
Member Author

CISC commented Mar 1, 2025

Failing tests seems to just be server outage.

@ngxson
Copy link
Copy Markdown
Contributor

ngxson commented Mar 1, 2025

I'll test this with Phi-4 a bit later

Comment thread examples/main/main.cpp Outdated
@ngxson
Copy link
Copy Markdown
Contributor

ngxson commented Mar 2, 2025

Ok that works, here is my test command + result (leaving it here for visibility):

No system prompt:

./build/bin/llama-cli -m phi-4-IQ2_M.gguf

 - Not using system message. To change it, set a different value via -sys PROMPT

> hi
Hello! How can I assist you today?

Incorrect use -p

./build/bin/llama-cli -m phi-4-IQ2_M.gguf -p test

*** User-specified prompt in conversation mode will be ignored, did you mean to set --system-prompt (-sys) instead?

 - Not using system message. To change it, set a different value via -sys PROMPT
...

Correctly using system prompt:

./build/bin/llama-cli -m phi-4-IQ2_M.gguf -sys "Your name is Sydney"

systemYour name is Sydney


> what is your name
My name is Sydney. How can I assist you today?

@CISC
Copy link
Copy Markdown
Member Author

CISC commented Mar 2, 2025

@ngxson I'll make a followup PR to add template formatting in non-conversation mode, and combined use of -sys and -p together.

@ngxson ngxson merged commit 14dec0c into ggml-org:master Mar 2, 2025
@CISC CISC deleted the jinja-system-prompt branch March 2, 2025 14:12
arthw pushed a commit to arthw/llama.cpp that referenced this pull request Mar 19, 2025
* Use jinja chat template system prompt by default

* faster conditional order

* remove nested ternary

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Seunghhon pushed a commit to Seunghhon/llama.cpp that referenced this pull request Apr 26, 2026
* Use jinja chat template system prompt by default

* faster conditional order

* remove nested ternary

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants