Skip to content

[breaking] [Tests] Large refactoring of Test #1404

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
odow wants to merge 23 commits into from
Closed

[breaking] [Tests] Large refactoring of Test #1404

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
f382b1b
[Tests] Large refactoring of Test/contlinear.jl
odow Jun 22, 2021
e2da047
Move the tests' tests into the same files and migrate UnitTests/varia…
odow Jun 22, 2021
c8726dc
Fix tests
odow Jun 22, 2021
6dc39d3
Migrate intconic
odow Jun 22, 2021
bcaa26e
Migrate intlinear
odow Jun 22, 2021
a887820
Fix formatting
odow Jun 22, 2021
4df1cdb
Migrate nlp.jl tests
odow Jun 22, 2021
12c5b6c
Fix tests
odow Jun 22, 2021
66dbc5d
Another fix
odow Jun 22, 2021
3c9cad5
Improve code coverage
odow Jun 22, 2021
c21c8cb
Migrate contquadratic
odow Jun 22, 2021
7c9a073
Migrate UnitTests/attributes.jl
odow Jun 22, 2021
2b35b16
Migrate UnitTests/modifications.jl
odow Jun 22, 2021
79263f1
Various fixes and updates to documentation
odow Jun 22, 2021
754410b
Rename tests in attributes.jl
odow Jun 22, 2021
528f115
Fix bridge tests
odow Jun 22, 2021
b19ba03
More migration of UnitTests
odow Jun 22, 2021
453a1f5
Refactoring of basic functions
odow Jun 22, 2021
1530835
Migrate UnitTests/constraints.jl
odow Jun 23, 2021
aede960
Fix formatting
odow Jun 23, 2021
1a15823
WIP: migrate contconic.jl
odow Jun 23, 2021
532b6f0
Fix formatting
odow Jun 23, 2021
6f8d581
WIP: continue migrating contconic
odow Jun 23, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
320 changes: 153 additions & 167 deletions docs/src/submodules/Test/overview.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,7 @@ so that all solvers can benefit.
## How to test a solver

The skeleton below can be used for the wrapper test file of a solver named
`FooBar`. Remove unnecessary tests as appropriate, for example tests for
features that the solver does not support (tests are not skipped depending
on the value of `supports`).
`FooBar`.

```julia
# ============================ /test/MOI_wrapper.jl ============================
Expand All @@ -34,15 +32,15 @@ using Test

const MOI = MathOptInterface

const OPTIMIZER_CONSTRUCTOR = MOI.OptimizerWithAttributes(
FooBar.Optimizer,
MOI.Silent() => true
const OPTIMIZER = MOI.instantiate(
MOI.OptimizerWithAttributes(FooBar.Optimizer, MOI.Silent() => true),
)
const OPTIMIZER = MOI.instantiate(OPTIMIZER_CONSTRUCTOR)

const BRIDGED = MOI.instantiate(
OPTIMIZER_CONSTRUCTOR, with_bridge_type = Float64
MOI.OptimizerWithAttributes(FooBar.Optimizer, MOI.Silent() => true),
with_bridge_type = Float64,
)

const CONFIG = MOI.Test.Config(
# Modify tolerances as necessary.
atol = 1e-6,
Expand All @@ -57,196 +55,184 @@ const CONFIG = MOI.Test.Config(
basis = false,
)

function test_SolverName()
@test MOI.get(OPTIMIZER, MOI.SolverName()) == "FooBar"
end
"""
runtests()

function test_supports_incremental_interface()
@test MOI.supports_incremental_interface(OPTIMIZER, false)
# Use `@test !...` if names are not supported
@test MOI.supports_incremental_interface(OPTIMIZER, true)
This function runs all functions in the this Module starting with `test_`.
"""
function runtests()
for name in names(@__MODULE__; all = true)
if startswith("$(name)", "test_")
@testset "$(name)" begin
getfield(@__MODULE__, name)()
end
end
end
end

function test_unittest()
# Test all the functions included in dictionary `MOI.Test.unittests`,
# except functions "number_threads" and "solve_qcp_edge_cases."
MOI.Test.unittest(
"""
test_runtests()

This function runs all the tests in MathOptInterface.Test.

Pass arguments to `exclude` to skip tests for functionality that is not
implemented or that your solver doesn't support.
"""
function test_runtests()
MOI.Test.runtests(
BRIDGED,
CONFIG,
["number_threads", "solve_qcp_edge_cases"]
exclude = [
"test_NumberOfThreads",
"solve_qcp_edge_cases",
]
)
return
end

function test_modification()
MOI.Test.modificationtest(BRIDGED, CONFIG)
end

function test_contlinear()
MOI.Test.contlineartest(BRIDGED, CONFIG)
end
"""
test_SolverName()

function test_contquadratictest()
MOI.Test.contquadratictest(OPTIMIZER, CONFIG)
You can also write new tests for solver-specific functionality. Write each new
test as a function with a name beginning with `test_`.
"""
function test_SolverName()
@test MOI.get(FooBar.Optimizer(), MOI.SolverName()) == "FooBar"
end

function test_contconic()
MOI.Test.contlineartest(BRIDGED, CONFIG)
end
end # module TestFooBar

function test_intconic()
MOI.Test.intconictest(BRIDGED, CONFIG)
end
# This line at tne end of the file runs all the tests!
Copy link
Copy Markdown
Member

@joaquimg joaquimg Jun 24, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tne -> the

TestFooBar.runtests()
```

function test_default_objective_test()
MOI.Test.default_objective_test(OPTIMIZER)
end
Then modify your `runtests.jl` file to include the `MOI_wrapper.jl` file:
```julia
# ============================ /test/runtests.jl ============================

function test_default_status_test()
MOI.Test.default_status_test(OPTIMIZER)
end
using Test

function test_nametest()
MOI.Test.nametest(OPTIMIZER)
@testset "MOI" begin
include("test/MOI_wrapper.jl")
end
```

function test_validtest()
MOI.Test.validtest(OPTIMIZER)
end
!!! info
The optimizer `BRIDGED` constructed with [`instantiate`](@ref)
automatically bridges constraints that are not supported by `OPTIMIZER`
using the bridges listed in [Bridges](@ref). It is recommended for an
implementation of MOI to only support constraints that are natively
supported by the solver and let bridges transform the constraint to the
appropriate form. For this reason it is expected that tests may not pass if
`OPTIMIZER` is used instead of `BRIDGED`.

function test_emptytest()
MOI.Test.emptytest(OPTIMIZER)
end
## How to add a test

function test_orderedindicestest()
MOI.Test.orderedindicestest(OPTIMIZER)
end
To detect bugs in solvers, we add new tests to `MOI.Test`.

function test_scalar_function_constant_not_zero()
MOI.Test.scalar_function_constant_not_zero(OPTIMIZER)
end
As an example, ECOS errored calling [`optimize!`](@ref) twice in a row. (See
[ECOS.jl PR #72](https://github.com/jump-dev/ECOS.jl/pull/72).) We could add a
test to ECOS.jl, but that would only stop us from re-introducing the bug to
ECOS.jl in the future, but it would not catch other solvers in the ecosystem
with the same bug! Instead, if we add a test to `MOI.Test`, then all solvers
will also check that they handle a double optimize call!

# This function runs all functions in this module starting with `test_`.
function runtests()
for name in names(@__MODULE__; all = true)
if startswith("$(name)", "test_")
@testset "$(name)" begin
getfield(@__MODULE__, name)()
end
end
end
end
For this test, we care about correctness, rather than performance. therefore, we
don't expect solvers to efficiently decide that they have already solved the
problem, only that calling [`optimize!`](@ref) twice doesn't throw an error or
give the wrong answer.

end # module TestFooBar
**Step 1**

TestFooBar.runtests()
Install the `MathOptInterface` julia package in `dev` mode
([ref](https://julialang.github.io/Pkg.jl/v1/managing-packages/#developing-1)):
```julia
julia> ]
(@v1.6) pkg> dev MathOptInterface
```

Test functions like `MOI.Test.unittest` and `MOI.Test.modificationtest` are
wrappers around corresponding dictionaries `MOI.Test.unittests` and
`MOI.Test.modificationtests`. Exclude tests by passing a vector of strings
corresponding to the test keys you want to exclude as the third positional
argument to the test function.
**Step 2**

!!! tip
Print a list of all keys using `println.(keys(MOI.Test.unittests))`

The optimizer `BRIDGED` constructed with [`instantiate`](@ref)
automatically bridges constraints that are not supported by `OPTIMIZER`
using the bridges listed in [Bridges](@ref). It is recommended for an
implementation of MOI to only support constraints that are natively supported
by the solver and let bridges transform the constraint to the appropriate form.
For this reason it is expected that tests may not pass if `OPTIMIZER` is used
instead of `BRIDGED`.

To test that a specific problem can be solved without bridges, a specific test
can be added with `OPTIMIZER` instead of `BRIDGED`. For example:
From here on, proceed with making the following changes in the
`~/.julia/dev/MathOptInterface` folder (or equivalent `dev` path on your
machine).

**Step 3**

Since the double-optimize error involves solving an optimization problem,
add a new test to [src/Test/UnitTests/solve.jl](https://github.com/jump-dev/MathOptInterface.jl/blob/master/src/Test/UnitTests/solve.jl).

The test should be something like
```julia
function test_interval_constraints()
MOI.Test.linear10test(OPTIMIZER, CONFIG)
"""
test_solve_twice(model::MOI.ModelLike, config::Config)

Test that calling `MOI.optimize!` twice does not error.

This problem was first detected in ECOS.jl PR#72:
https://github.com/jump-dev/ECOS.jl/pull/72
"""
function test_solve_twice(
model::MOI.ModelLike,
config::Config{T},
) where {T}
if !config.solve
# Use `config` to modify the behavior of the tests. Since this test is
# concerned with `optimize!`, we should skip the test if
# `config.solve == false`.
return
end
# This is important! Make sure to empty the existing model!
MOI.empty!(model)
# Create a simple model. Try to make this as simple as possible so that the
# majority of solvers can run the test.
x = MOI.add_variable(model)
MOI.add_constraint(model, MOI.SingleVariable(x), MOI.GreaterThan(one(T)))
MOI.set(model, MOI.ObjectiveSense(), MOI.MIN_SENSE)
MOI.set(
model,
MOI.ObjectiveFunction{MOI.SingleVariable}(),
MOI.SingleVariable(x),
)
# The main component of the test: does calling `optimize!` twice error?
MOI.optimize!(model)
MOI.optimize!(model)
# Check we have a solution.
MOI.get(model, MOI.TerminationStatus()) == MOI.OPTIMAL
MOI.get(model, MOI.VariablePrimal(), x) == one(T)
return
end
```
checks that `OPTIMIZER` implements support for
[`ScalarAffineFunction`](@ref)-in-[`Interval`](@ref).

## How to add a test

To give an example, ECOS errored calling [`optimize!`](@ref) twice in a row.
(See [ECOS.jl PR #72](https://github.com/jump-dev/ECOS.jl/pull/72).)
!!! info
Make sure the function is agnoistic to the number type `T`! Don't assume it
is a `Float64` capable solver!

We could add a test to ECOS.jl, but that would only stop us from re-introducing
the bug to ECOS.jl in the future.
We also need to write a test for the test. Place this function immediately below
the test you just wrote in the same file:
```julia
function setup_test(
::typeof(test_solve_twice),
model::MOI.Utilities.MockOptimizer,
config::Config,
)
MOI.Utilities.set_mock_optimize!(
model,
(mock::MOI.Utilities.MockOptimizer) -> MOIU.mock_optimize!(
mock,
MOI.OPTIMAL,
(MOI.FEASIBLE_POINT, [1.0]),
),
)
return
end
```

Instead, if we add a test to `MOI.Test`, then all solvers will also check that
they handle a double optimize call!
**Step 6**

For this test, we care about correctness, rather than performance. therefore, we
don't expect solvers to efficiently decide that they have already solved the
problem, only that calling [`optimize!`](@ref) twice doesn't throw an error or
give the wrong answer.
Commit the changes to git from `~/.julia/dev/MathOptInterface` and
submit the PR for review.

To resolve this issue, follow these steps (tested on Julia v1.5):

1. Install the `MathOptInterface` julia package in `dev` mode
([ref](https://julialang.github.io/Pkg.jl/v1/managing-packages/#developing-1)):
```julia
julia> ]
(@v1.5) pkg> dev ECOS
(@v1.5) pkg> dev MathOptInterface
```
2. From here on, proceed with making the following changes in the
`~/.julia/dev/MathOptInterface` folder (or equivalent `dev` path on your
machine).
3. Since the double-optimize error involves solving an optimization problem,
add a new test to [src/Test/UnitTests/solve.jl](https://github.com/jump-dev/MathOptInterface.jl/blob/master/src/Test/UnitTests/solve.jl).
The test should be something like
```julia
function solve_twice(model::MOI.ModelLike, config::Config)
MOI.empty!(model)
x = MOI.add_variable(model)
c = MOI.add_constraint(model, MOI.SingleVariable(x), MOI.GreaterThan(1.0))
MOI.set(model, MOI.ObjectiveSense(), MOI.MIN_SENSE)
MOI.set(model, MOI.ObjectiveFunction{MOI.SingleVariable}(), MOI.SingleVariable(x))
if config.solve
MOI.optimize!(model)
MOI.optimize!(model)
MOI.get(model, MOI.TerminationStatus()) == MOI.OPTIMAL
MOI.get(model, MOI.VariablePrimal(), x) == 1.0
end
end
unittests["solve_twice"] = solve_twice
```
2. Add a test for the test you just wrote. (We test the tests!)
a. Add the name of the test (`"solve_twice"`) to the end of the array in
`MOI.Test.unittest(...)` ([link](https://github.com/jump-dev/MathOptInterface.jl/blob/7543afe4b5151cf36bbd18181c1bb5c83266ae2f/test/Test/unit.jl#L51-L52)).
b. Add a test for the test towards the end of the "Unit Tests" test set
([link](https://github.com/jump-dev/MathOptInterface.jl/blob/7543afe4b5151cf36bbd18181c1bb5c83266ae2f/test/Test/unit.jl#L394)).
The test should look something like
```julia
@testset "solve_twice" begin
MOI.Utilities.set_mock_optimize!(mock,
(mock::MOI.Utilities.MockOptimizer) -> MOI.Utilities.mock_optimize!(
mock,
MOI.OPTIMAL,
(MOI.FEASIBLE_POINT, [1.0]),
),
(mock::MOI.Utilities.MockOptimizer) -> MOI.Utilities.mock_optimize!(
mock,
MOI.OPTIMAL,
(MOI.FEASIBLE_POINT, [1.0]),
)
)
MOI.Test.solve_twice(mock, config)
end
```
In the above `mock` is a `MOI.Utilities.MockOptimizer` that is defined
tesearlier in the file. In this test, `MOI.Utilities.set_mock_optimize!` loads
`mock` with two results. Each says that the
[`TerminationStatus`](@ref) is `MOI.OPTIMAL`, that the
[`PrimalStatus`](@ref) is `MOI.FEASIBLE_POINT`, and that there is one
variable with a `MOI.VariableValue` or `1.0`
3. Run the tests:
```julia
(@v1.5) pkg> test ECOS
```
4. Finally, commit the changes to git from `~/.julia/dev/MathOptInterface` and
submit the PR for review.
!!! tip
If you need help writing a test, [open an issue on GitHub](https://github.com/jump-dev/MathOptInterface.jl/issues/new),
or ask the [Developer Chatroom](https://gitter.im/JuliaOpt/JuMP.jl)
Loading