Skip to content

server: fix passing prompt as tokens#5955

Merged
ggerganov merged 2 commits intoggml-org:masterfrom
z80maniac:fix-prompt-as-tokens
Mar 9, 2024
Merged

server: fix passing prompt as tokens#5955
ggerganov merged 2 commits intoggml-org:masterfrom
z80maniac:fix-prompt-as-tokens

Conversation

@z80maniac
Copy link
Copy Markdown
Contributor

This PR fixes a crash in server that happens if prompt is an array of tokens.

Before:

❯ curl -sS --data '{"prompt": [15043,29892,590,1024,338], "n_predict": 4}' http://127.0.0.1:8080/completion | jq .content
curl: (52) Empty reply from server
terminate called after throwing an instance of 'nlohmann::json_abi_v3_11_2::detail::type_error'
  what():  [json.exception.type_error.302] type must be string, but is array
fish: Job 1, './server -m /opt/models/text/ll…' terminated by signal SIGABRT (Abort)

After:

❯ curl -sS --data '{"prompt": [15043,29892,590,1024,338], "n_predict": 4}' http://127.0.0.1:8080/completion | jq .content
" John and I have"

Comment thread examples/server/server.cpp
@ggerganov ggerganov merged commit 0db32be into ggml-org:master Mar 9, 2024
@phymbert
Copy link
Copy Markdown
Collaborator

phymbert commented Mar 9, 2024

it would be nice to add a test on that feature. Else it will probably disappear again in the future

hazelnutcloud pushed a commit to hazelnutcloud/llama.cpp that referenced this pull request Mar 10, 2024
* server: fix passing prompt as tokens

* Update examples/server/server.cpp

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
NeoZhangJianyu pushed a commit to NeoZhangJianyu/llama.cpp that referenced this pull request Mar 12, 2024
* server: fix passing prompt as tokens

* Update examples/server/server.cpp

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
jordankanter pushed a commit to jordankanter/llama.cpp that referenced this pull request Mar 13, 2024
* server: fix passing prompt as tokens

* Update examples/server/server.cpp

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Seunghhon pushed a commit to Seunghhon/llama.cpp that referenced this pull request Apr 26, 2026
* server: fix passing prompt as tokens

* Update examples/server/server.cpp

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
phuongncn pushed a commit to phuongncn/llama.cpp-gx10-dgx-sparks-deepseekv4 that referenced this pull request Apr 28, 2026
* server: fix passing prompt as tokens

* Update examples/server/server.cpp

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants