UT - resolver " FAIL: TestSolveOperators_WithSystemConstraints"

[akihikokuroda]

Bug Report

Unit test - resolver_test.go " FAIL: TestSolveOperators_WithSystemConstraints"
https://github.com/operator-framework/operator-lifecycle-manager/runs/4756877867?check_suite_focus=true#step:5:53

--- FAIL: TestSolveOperators_WithSystemConstraints (0.01s)
    resolver_test.go:164: 
        	Error Trace:	resolver_test.go:164
        	Error:      	"constraints not satisfiable: package: packageC is not white listed, package: packageB is not white listed, subscription packageA-alpha requires test-catalog/test-namespace/alpha/packageA.v1, subscription packageA-alpha exists, bundle packageA.v1 requires an operator providing an API with group: g, version: v, kind: k" does not contain "packageA is not white listed"
        	Test:       	TestSolveOperators_WithSystemConstraints
        	Messages:   	Test Runtime constraints don't allow A failed
time="2022-01-10T03:41:22Z" level=info msg="considered csvs without properties annotation during resolution: [packageA.v1]"
time="2022-01-10T03:41:22Z" level=warning msg="could not unambiguously infer package name for \"packageA.v1\" (found 0 distinct package names)"
time="2022-01-10T03:41:22Z" level=info msg="considered csvs without properties annotation during resolution: [packageA.v1]"
time="2022-01-10T03:41:22Z" level=warning msg="could not unambiguously infer package name for \"packageA.v1\" (found 0 distinct package names)"
time="2022-01-10T03:41:22Z" level=info msg="considered csvs without properties annotation during resolution: [packageA.v1]"
time="2022-01-10T03:41:22Z" level=warning msg="could not unambiguously infer package name for \"packageA.v1\" (found 0 distinct package names)"
time="2022-01-10T03:41:22Z" level=info msg="considered csvs without properties annotation during resolution: [packageA.v1]"
time="2022-01-10T03:41:22Z" level=warning msg="could not unambiguously infer package name for \"packageA.v1\" (found 0 distinct package names)"
time="2022-01-10T03:41:22Z" level=info msg="considered csvs without properties annotation during resolution: [packageA.v1]"
time="2022-01-10T03:41:22Z" level=info msg="considered csvs without properties annotation during resolution: [packageA.v1]"
time="2022-01-10T03:41:22Z" level=warning msg="could not unambiguously infer package name for \"packageA.v1\" (found 0 distinct package names)"
time="2022-01-10T03:41:22Z" level=info msg="considered csvs without properties annotation during resolution: [packageA.v1]"
time="2022-01-10T03:41:22Z" level=info msg="considered csvs without properties annotation during resolution: [packageA.v1]"
time="2022-01-10T03:41:22Z" level=warning msg="could not unambiguously infer package name for \"packageA.v1\" (found 0 distinct package names)"
time="2022-01-10T03:41:22Z" level=info msg="considered csvs without properties annotation during resolution: [packageA.v1]"
time="2022-01-10T03:41:22Z" level=warning msg="could not unambiguously infer package name for \"packageA.v1\" (found 0 distinct package names)"
time="2022-01-10T03:41:22Z" level=info msg="considered csvs without properties annotation during resolution: [packageA.v1]"
time="2022-01-10T03:41:22Z" level=warning msg="could not unambiguously infer package name for \"packageA.v1\" (found 0 distinct package names)"
time="2022-01-10T03:41:22Z" level=info msg="considered csvs without properties annotation during resolution: [packageA.v1]"
time="2022-01-10T03:41:22Z" level=warning msg="could not unambiguously infer package name for \"packageA.v1\" (found 0 distinct package names)"
time="2022-01-10T03:41:22Z" level=info msg="considered csvs without properties annotation during resolution: [packageA.v1]"
time="2022-01-10T03:41:22Z" level=info msg="considered csvs without properties annotation during resolution: [opA.v1.0.0 opB.v1.0.0]"
time="2022-01-10T03:41:23Z" level=info msg="considered csvs without properties annotation during resolution: [a.v1]"
time="2022-01-10T03:41:24Z" level=info msg="considered csvs without properties annotation during resolution: [a.v1]"
constraints not satisfiable: subscription a-alpha requires catsrc/catsrc-namespace/alpha/a.v1, subscription a-alpha exists, catsrc/catsrc-namespace/alpha/a.v1 and catsrc/catsrc-namespace/beta/a.v2 originate from package a, subscription a-beta requires catsrc/catsrc-namespace/beta/a.v2, subscription a-beta exists
time="2022-01-10T03:41:24Z" level=info msg="considered csvs without properties annotation during resolution: [a.v1]"
time="2022-01-10T03:41:24Z" level=warning msg="could not unambiguously infer package name for \"a.v1\" (found 0 distinct package names)"
time="2022-01-10T03:41:24Z" level=info msg="considered csvs without properties annotation during resolution: [a.v1]"
time="2022-01-10T03:41:24Z" level=info msg="considered csvs without properties annotation during resolution: [a.v1]"
time="2022-01-10T03:41:24Z" level=info msg="considered csvs without properties annotation during resolution: [a.v1]"
time="2022-01-10T03:41:24Z" level=info msg="considered csvs without properties annotation during resolution: [a.v1]"
time="2022-01-10T03:41:25Z" level=warning msg="could not unambiguously infer package name for \"a.v1\" (found 0 distinct package names)"
time="2022-01-10T03:41:25Z" level=warning msg="could not unambiguously infer package name for \"b.v1\" (found 0 distinct package names)"
time="2022-01-10T03:41:25Z" level=info msg="considered csvs without properties annotation during resolution: [a.v1 b.v1]"
time="2022-01-10T03:41:25Z" level=warning msg="could not unambiguously infer package name for \"a.v1\" (found 0 distinct package names)"
time="2022-01-10T03:41:25Z" level=warning msg="could not unambiguously infer package name for \"b.v1\" (found 0 distinct package names)"
time="2022-01-10T03:41:25Z" level=info msg="considered csvs without properties annotation during resolution: [a.v1 b.v1]"
time="2022-01-10T03:41:25Z" level=warning msg="could not unambiguously infer package name for \"a.v1\" (found 0 distinct package names)"
time="2022-01-10T03:41:25Z" level=info msg="considered csvs without properties annotation during resolution: [a.v1]"
time="2022-01-10T03:41:25Z" level=info msg="considered csvs without properties annotation during resolution: [a.v1 b.v1]"
time="2022-01-10T03:41:25Z" level=warning msg="could not unambiguously infer package name for \"a.v1\" (found 0 distinct package names)"
time="2022-01-10T03:41:25Z" level=info msg="considered csvs without properties annotation during resolution: [a.v1]"
time="2022-01-10T03:41:25Z" level=warning msg="could not unambiguously infer package name for \"a.v1\" (found 0 distinct package names)"
time="2022-01-10T03:41:25Z" level=info msg="considered csvs without properties annotation during resolution: [a.v1]"
FAIL
FAIL	github.com/operator-framework/operator-lifecycle-manager/pkg/controller/registry/resolver	3.763s

What did you do?
CI Unit test intermittently fails at TestSolveOperators_WithSystemConstraints.

What did you expect to see?
No failure

What did you see instead? Under which circumstances?
Failure above.

Environment

• operator-lifecycle-manager version:

• Kubernetes version information:

• Kubernetes cluster kind:

Possible Solution

Additional context
Add any other context about the problem here.

[perdasilva]

I can take a look at this. I believe I might have introduced this somehow.

[perdasilva]

I've spent a little time on this but couldn't pin point exactly the source of the non-determinism other than to say it's coming from the solver. It's easy to (eventually) reproduce the bug by simply wrapping the test code around a loop and executing it. The test fails usually withing 10 executions.

I've tried to figure out whether the order of the inputs was the source of the issues, since that's created from map and therefore could result in different order. Sorting the inputs by name did not result in a deterministic result. It just seems like sometimes the resolver fails for one reason, other times for a different reason.

Since this isn't on any critical path yet, let's merge Aki's mitigation for now to (for CI's sake) and return to this when we want to invest more time on this feature. I'm tracking this internally on: https://issues.redhat.com/browse/OLM-2414 under the runtime constraints epic.