Skip to content

Handle server exception on wrong type in request#6452

Merged
ggerganov merged 1 commit intoggml-org:masterfrom
JH23X:server-handle-type-mismatch
Apr 3, 2024
Merged

Handle server exception on wrong type in request#6452
ggerganov merged 1 commit intoggml-org:masterfrom
JH23X:server-handle-type-mismatch

Conversation

@JH23X
Copy link
Copy Markdown
Contributor

@JH23X JH23X commented Apr 3, 2024

Currently the http server shuts down when an argument is supplied as a wrong type in the request json. For example,
{"prompt": "Building a website can be done in 10 simple steps:","n_predict": 128, "stream": "false"}' (notice the "false" that is supplied as a string)
will cause the following exception on the server:

terminate called after throwing an instance of 'nlohmann::json_abi_v3_11_3::detail::type_error' {"tid": ...}
what(): [json.exception.type_error.302] type must be boolean, but is string
Aborted

The server shouldn't shut down from a malformed request. The proposed change catches this exception and logs a warning instead:
{"tid": ... ,"level":"WARN","function":"json_value","line":64,msg:"Wrong type supplied for parameter 'stream'. Expected 'b', using default value.","prompt":"Building a website can be done in 10 simple steps:","n_predict": 128,"stream":"false"}

Not 100% sure if this is the right way to go about this, so feel free to let me know of any changes you would like to see!

@github-actions

This comment was marked as off-topic.

Copy link
Copy Markdown
Collaborator

@phymbert phymbert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, please wait for @ngxson review

@phymbert phymbert requested a review from ggerganov April 3, 2024 14:07
@ggerganov ggerganov merged commit 60cdf40 into ggml-org:master Apr 3, 2024
Comment thread examples/server/utils.hpp
}
catch (nlohmann::json_abi_v3_11_3::detail::type_error const&){
std::string message = "Wrong type supplied for parameter '" + key + "'. Expected '" + typeid(default_value).name() + "', using default value.";
server_log("WARN", __func__, __LINE__, message.c_str(), body);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can simply use LOG_WARNING here instead of calling directly server_log. But it's just a small detail, not important.

Seunghhon pushed a commit to Seunghhon/llama.cpp that referenced this pull request Apr 26, 2026
Co-authored-by: Jonas Holzner <jonas.holzner.external@hensoldt.net>
phuongncn pushed a commit to phuongncn/llama.cpp-gx10-dgx-sparks-deepseekv4 that referenced this pull request Apr 28, 2026
Co-authored-by: Jonas Holzner <jonas.holzner.external@hensoldt.net>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants