Skip to content

Blacklist test instead of stopping if bench fails.#4

Open
Menkib64 wants to merge 1 commit intoLeelaChessZero:masterfrom
Menkib64:bench_fail_blacklist
Open

Blacklist test instead of stopping if bench fails.#4
Menkib64 wants to merge 1 commit intoLeelaChessZero:masterfrom
Menkib64:bench_fail_blacklist

Conversation

@Menkib64
Copy link
Copy Markdown

There is fairly common case that a worker has GPU problems. GPU problems will cause a bench failure. Reporting of bench failure stops the test. This is different to how compile errors are handled using client blacklist. This change makes bench error handle the same as compile errors.

There is fairly common case that a worker has GPU problems. GPU problems
will cause a bench failure. Reporting of bench failure stops the test.
This is different to how compile errors are handled using client
blacklist. This change makes bench error handle the same as compile
errors.
Copilot AI review requested due to automatic review settings April 25, 2026 13:09
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Aligns benchmark-failure handling with compile/build-failure handling by blacklisting the workload on the client rather than stopping the entire test on the server.

Changes:

  • Server: client_bench_error no longer marks the test as finished; it only logs the bench error event.
  • Client: on OpenBenchBadBenchException, the worker now appends the current test.id to config.blacklist before reporting the bench error.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
OpenBench/views.py Removes server-side “stop test” behavior on bench error; keeps logging the event.
Client/worker.py Adds the failing test id to the client blacklist on bench failure to avoid re-assignment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants