llama-server: add router multi-model tests (#17704)

[ServeurpersoCom]

Add 4 test cases for model router:

• test_router_unload_model: explicit model unloading
• test_router_models_max_evicts_lru: LRU eviction with --models-max
• test_router_no_models_autoload: --no-models-autoload flag behavior
• test_router_api_key_required: API key authentication

Tests use async model loading with polling and graceful skip when insufficient models available for eviction testing.

utils.py changes:

• Add models_max, models_dir, no_models_autoload attributes to ServerProcess
• Handle JSONDecodeError for non-JSON error responses (fallback to text)

Make sure to read the contributing guidelines before submitting a PR

Close #17704

[ngxson]

I mirrored the PR here for faster CI: ngxson#48

[ngxson]

Great jobs, thanks! Merging once CI passes

[ServeurpersoCom]

On empty tmp/* I check this after eating :

FAILED unit/test_router.py::test_router_models_max_evicts_lru - AssertionError: Timed out waiting for ggml-org/tinygemma3-GGUF:Q8_0 to reach {'loaded'}, last status: unloaded

[ngxson]

@ServeurpersoCom that can happen if ServerPreset.load_all() was never called. We had a fixture to automatically call it before any tests start, but maybe it's not called in your case?

[ServeurpersoCom]

@ServeurpersoCom that can happen if ServerPreset.load_all() was never called. We had a fixture to automatically call it before any tests start, but maybe it's not called in your case?

Yes. Missing. I need this :

import pytest
from utils import ServerPreset

@pytest.fixture(scope="module", autouse=True)
def ensure_models_cached():
"""Ensure all models are cached before router tests"""
ServerPreset.load_all()

[ngxson]

Hmm not sure why we're getting IP rate-limited by HF. Can you make sure that the offline option is set for all models?

The idea is that load_all() will be called once just to download models to cache, then tests will run with offline = True to avoid sending too many requests to HF server

[ServeurpersoCom]

I try also :

(root|~/llama.cpp.pascal) git diff
diff --git a/tools/server/tests/unit/test_router.py b/tools/server/tests/unit/test_router.py
index 87710d511..fb304386c 100644
--- a/tools/server/tests/unit/test_router.py
+++ b/tools/server/tests/unit/test_router.py
@@ -3,6 +3,11 @@ from utils import *

 server: ServerProcess

+@pytest.fixture(scope="module", autouse=True)
+def preload_models():
+    # Preload all models before running router tests
+    ServerPreset.load_all()
+
 @pytest.fixture(autouse=True)
 def create_server():
     global server
@@ -17,7 +22,6 @@ def create_server():
     ]
 )
 def test_router_chat_completion_stream(model: str, success: bool):
-    # TODO: make sure the model is in cache (ie. ServerProcess.load_all()) before starting the router server
     global server
     server.start()
     content = ""
(root|~/llama.cpp.pascal) ./test.sh

[ngxson]

I mean, the scope="module" is suppose to be outside of the test file, as it will technically affect all test units.

The fixture is already added to conftest.py

[ServeurpersoCom]

OK I found the problem

[ServeurpersoCom]

(root|~/llama.cpp.pascal) ./test.sh
===================================================================== test session starts ======================================================================
platform linux -- Python 3.11.2, pytest-8.3.5, pluggy-1.6.0 -- /root/llama.cpp.pascal/tools/server/tests/.venv/bin/python3
cachedir: .pytest_cache
rootdir: /root/llama.cpp.pascal/tools/server/tests
configfile: pytest.ini
plugins: anyio-4.12.0
collected 6 items

unit/test_router.py::test_router_chat_completion_stream[ggml-org/tinygemma3-GGUF:Q8_0-True] PASSED                                                       [ 16%]
unit/test_router.py::test_router_chat_completion_stream[non-existent/model-False] PASSED                                                                 [ 33%]
unit/test_router.py::test_router_unload_model PASSED                                                                                                     [ 50%]
unit/test_router.py::test_router_models_max_evicts_lru PASSED                                                                                            [ 66%]
unit/test_router.py::test_router_no_models_autoload PASSED                                                                                               [ 83%]
unit/test_router.py::test_router_api_key_required PASSED                                                                                                 [100%]

====================================================================== 6 passed in 15.92s ======================================================================

[ngxson]

Can be merged once the CI ngxson#48 passes

[ngxson]

Hmm, seems like on windows, the fixture isn't called: https://github.com/ngxson/llama.cpp/actions/runs/19895083434/job/57023680373?pr=48

Maybe there is a problem with multi-shard model?

[ngxson]

If the problem still persist with multi-shard model, we can remove the offline for multi-shard model, because it is only run used once anyway.

We can add a TODO and fix the problem later (not very important)

Edit: ah, nevermind, your fix seems to be OK!