Skip to content

Hard to debug failed upgrade tests #3538

@vaikas

Description

@vaikas

We have a few tests that are failing with something like this:

==============================
==== STARTING PROBER TEST ====
==============================
==== Tue Jul  7 14:11:02 PDT 2020
==============================
Prober PID is 48353
Waiting for existance of file: /tmp/prober-ready
Running tests with 'go test -v  -race -count=1 -tags=probe -timeout=30m ./test/upgrade --pipefile=/tmp/prober-signal --readyfile=/tmp/prober-ready '
=== RUN   TestContinuousEventsPropagationWithProber
    TestContinuousEventsPropagationWithProber: test_runner.go:99: namespace is : "test-continuous-events-propagation-with-prober-t74wg"
2020-07-07T21:11:10.472Z	INFO	prober/prober.go:134	Using namespace for probe testing: test-continuous-events-propagation-with-prober-t74wg
2020-07-07T21:11:10.486Z	INFO	prober/configuration.go:132	Deploying trigger: wathola-trigger-step
2020-07-07T21:11:10.508Z	INFO	prober/configuration.go:132	Deploying trigger: wathola-trigger-finished
2020-07-07T21:11:10.521Z	INFO	prober/receiver.go:46	Deploy of receiver pod: wathola-receiver
2020-07-07T21:11:10.711Z	INFO	prober/receiver.go:98	Deploy of receiver service: wathola-receiver
2020-07-07T21:11:10.742Z	DEBUG	prober/receiver.go:133	Node port for service: wathola-receiver is 31799
2020-07-07T21:11:10.743Z	INFO	lib/await.go:44	Wait for configmap be deployed: wathola-config
2020-07-07T21:11:10.743Z	INFO	prober/configuration.go:90	Deploying config map: wathola-config
2020-07-07T21:11:10.743Z	DEBUG	prober/configuration.go:72	Fetching default broker URL for ns test-continuous-events-propagation-with-prober-t74wg
2020-07-07T21:11:10.743Z	INFO	lib/await.go:44	Wait for trigger be ready: wathola-trigger-step
2020-07-07T21:11:10.744Z	INFO	lib/await.go:44	Wait for receiver be ready: wathola-receiver
2020-07-07T21:11:10.744Z	INFO	lib/await.go:44	Wait for trigger be ready: wathola-trigger-finished
2020-07-07T21:13:10.764Z	ERROR	lib/await.go:49	Error while waiting for configmap be deployed: wathola-config: timed out waiting for the condition
knative.dev/eventing/test/lib.AwaitForAll.func1
	/home/prow/go/src/knative.dev/eventing/test/lib/await.go:49
golang.org/x/sync/errgroup.(*Group).Go.func1
	/home/prow/go/src/knative.dev/eventing/vendor/golang.org/x/sync/errgroup/errgroup.go:57
2020-07-07T21:13:10.767Z	ERROR	lib/await.go:49	Error while waiting for trigger be ready: wathola-trigger-step: timed out waiting for the condition
knative.dev/eventing/test/lib.AwaitForAll.func1
	/home/prow/go/src/knative.dev/eventing/test/lib/await.go:49
golang.org/x/sync/errgroup.(*Group).Go.func1
	/home/prow/go/src/knative.dev/eventing/vendor/golang.org/x/sync/errgroup/errgroup.go:57
2020-07-07T21:13:10.771Z	ERROR	lib/await.go:49	Error while waiting for trigger be ready: wathola-trigger-finished: timed out waiting for the condition
knative.dev/eventing/test/lib.AwaitForAll.func1
	/home/prow/go/src/knative.dev/eventing/test/lib/await.go:49
golang.org/x/sync/errgroup.(*Group).Go.func1
	/home/prow/go/src/knative.dev/eventing/vendor/golang.org/x/sync/errgroup/errgroup.go:57
ERROR: Timeout (300s) while waiting for the file /tmp/prober-ready.

It's not possible to understand from here if the trigger was created, what state it is and so forth. For e2e tests, they dump the state of the resources they have created at the end (as well as k8s events), we should have that kind of information available for these tests as well. Or if it's already there, maybe the way we exit, they the resources dumper / events dumper doesn't run?

@cardil would you mind taking a look?

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugCategorizes issue or PR as related to a bug.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions