From 0dd49b12b1eb364420238314fe9c7f0b57ac67c6 Mon Sep 17 00:00:00 2001 From: Oscar Dowson Date: Thu, 15 Jul 2021 11:41:32 +1200 Subject: [PATCH 1/3] [Test] document how to debug a failing test --- docs/src/submodules/Test/overview.md | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/docs/src/submodules/Test/overview.md b/docs/src/submodules/Test/overview.md index c313eefc70..8e03f73cb4 100644 --- a/docs/src/submodules/Test/overview.md +++ b/docs/src/submodules/Test/overview.md @@ -48,6 +48,9 @@ const CONFIG = MOI.Test.Config( rtol = 1e-6, # Use MOI.LOCALLY_SOLVED for local solvers. optimal_status = MOI.OPTIMAL, + # Pass attributes or MOI functions to `exclude` to skip tests that + # rely on this functionality. + exclude = Any[MOI.VariableName, MOI.delete], ) """ @@ -122,6 +125,28 @@ end appropriate form. For this reason it is expected that tests may not pass if `OPTIMIZER` is used instead of `BRIDGED`. +## How to debug a failing test + +When writing a solver, it's likely that you will initially fail many tests! +Some failures will be bugs, but other failure you may choose to exclude. + +Add excludes by either passing a string to `MOI.Test.runtests`, or by passing +attributes or functions to `MOI.Test.Config`. + +Each test that fails can be independently called as: +```julia +model = FooBar.Optimizer() +config = MOI.Test.Config() +MOI.empty!(model) +MOI.Test.test_category_name_that_failed(model, config) +``` + +You can look-up the source code of the test that failed by searching for it +in the `src/Test/test_category.jl` file. + +Each test function also has a docstring that explains what the test is +for: `? MOI.Test.test_category_name_that_failed`. + ## How to add a test To detect bugs in solvers, we add new tests to `MOI.Test`. From 5fc262c4c27d96d8d1b82890a0b187acbf3548c3 Mon Sep 17 00:00:00 2001 From: Oscar Dowson Date: Thu, 15 Jul 2021 11:55:22 +1200 Subject: [PATCH 2/3] Update overview.md --- docs/src/submodules/Test/overview.md | 25 +++++++++++++++++++++---- 1 file changed, 21 insertions(+), 4 deletions(-) diff --git a/docs/src/submodules/Test/overview.md b/docs/src/submodules/Test/overview.md index 8e03f73cb4..1e3ca7275b 100644 --- a/docs/src/submodules/Test/overview.md +++ b/docs/src/submodules/Test/overview.md @@ -128,10 +128,27 @@ end ## How to debug a failing test When writing a solver, it's likely that you will initially fail many tests! -Some failures will be bugs, but other failure you may choose to exclude. - -Add excludes by either passing a string to `MOI.Test.runtests`, or by passing -attributes or functions to `MOI.Test.Config`. +Some failures will be bugs, but other failures you may choose to exclude. + +There are two ways to exclude tests: + + - Exclude tests whose names contain a string using: + ```julia + MOI.Test.runtests( + model, + config; + exclude = String["test_to_exclude", "test_conic_"], + ) + ``` + This will exclude tests whose name contains either of the two strings + provided. + + - Exclude tests which rely on specific functionality using: + ```julia + MOI.Test.Config(exclude = Any[MOI.VariableName, MOI.optimize!]) + ``` + This will exclude tests which use the `MOI.VariableName` attribute, + or which call `MOI.optimize!`. Each test that fails can be independently called as: ```julia From 8e2c973982d85fb5470b6c3efd41ea85110e71a1 Mon Sep 17 00:00:00 2001 From: Oscar Dowson Date: Thu, 15 Jul 2021 11:56:12 +1200 Subject: [PATCH 3/3] Update overview.md --- docs/src/submodules/Test/overview.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/src/submodules/Test/overview.md b/docs/src/submodules/Test/overview.md index 1e3ca7275b..8d13b25a32 100644 --- a/docs/src/submodules/Test/overview.md +++ b/docs/src/submodules/Test/overview.md @@ -161,8 +161,10 @@ MOI.Test.test_category_name_that_failed(model, config) You can look-up the source code of the test that failed by searching for it in the `src/Test/test_category.jl` file. -Each test function also has a docstring that explains what the test is -for: `? MOI.Test.test_category_name_that_failed`. +!!! tip + Each test function also has a docstring that explains what the test is + for. Use `? MOI.Test.test_category_name_that_failed` from the REPL to + read it. ## How to add a test