Fail in goroutine after TestBrokerAsMiddleware/Setup has completed

[pierDipi]

Describe the bug

Upgrading dependencies brings a bug in TestBrokerAsMiddleware: #4622

=== RUN   TestBrokerAsMiddleware/Assert/[Stable/MUST]broker_as_middleware_deliver_an_event
=== PAUSE TestBrokerAsMiddleware/Assert/[Stable/MUST]broker_as_middleware_deliver_an_event
=== CONT  TestBrokerAsMiddleware/Assert/[Stable/MUST]broker_as_middleware_deliver_an_event
2020/12/08 01:55:33 eventing.knative.dev/v1, Resource=brokers customname is addressable: http://broker-ingress.knative-eventing-e4mmywinri.svc.cluster.local/rekt-jwtefgkk/customname
        --- FAIL: TestBrokerAsMiddleware/Assert/[Stable/MUST]broker_as_middleware_deliver_an_event (240.00s)
panic: Fail in goroutine after TestBrokerAsMiddleware/Setup has completed [recovered]
	panic: Fail in goroutine after TestBrokerAsMiddleware/Setup has completed

goroutine 357 [running]:
testing.tRunner.func1.1(0x2277c40, 0xc00019a8a0)
	/root/.gvm/gos/go1.14.12/src/testing/testing.go:999 +0x461
testing.tRunner.func1(0xc000b6e120)
	/root/.gvm/gos/go1.14.12/src/testing/testing.go:1002 +0x606
panic(0x2277c40, 0xc00019a8a0)
	/root/.gvm/gos/go1.14.12/src/runtime/panic.go:975 +0x3e3
testing.(*common).Fail(0xc0005d77a0)
	/root/.gvm/gos/go1.14.12/src/testing/testing.go:671 +0x1fc
testing.(*common).Fail(0xc0005d78c0)
	/root/.gvm/gos/go1.14.12/src/testing/testing.go:665 +0x156
testing.(*common).FailNow(0xc0005d78c0)
	/root/.gvm/gos/go1.14.12/src/testing/testing.go:693 +0x39
testing.(*common).Fatalf(0xc0005d78c0, 0x25b0a5a, 0x33, 0xc00000e260, 0x2, 0x2)
	/root/.gvm/gos/go1.14.12/src/testing/testing.go:789 +0x9e
knative.dev/reconciler-test/pkg/eventshub.(*Store).AssertAtLeast(0xc0005e1f80, 0x1, 0xc000962358, 0x1, 0x1, 0x38f69c8, 0x3e, 0xc0006962c8)
	/home/prow/go/src/knative.dev/eventing/vendor/knative.dev/reconciler-test/pkg/eventshub/event_info_store.go:159 +0x323
knative.dev/reconciler-test/pkg/eventshub.(*Store).AssertInRange(0xc0005e1f80, 0x1, 0x1, 0xc000962358, 0x1, 0x1, 0xc0000b2cd0, 0x43f548, 0x8)
	/home/prow/go/src/knative.dev/eventing/vendor/knative.dev/reconciler-test/pkg/eventshub/event_info_store.go:169 +0xb1
knative.dev/reconciler-test/pkg/eventshub.(*Store).AssertExact(0xc0005e1f80, 0x1, 0xc000962358, 0x1, 0x1, 0x3d, 0xc000938e08, 0x3d)
	/home/prow/go/src/knative.dev/eventing/vendor/knative.dev/reconciler-test/pkg/eventshub/event_info_store.go:199 +0x99
knative.dev/eventing/test/rekt/features.BrokerAsMiddleware.func2(0x2a8ff40, 0xc0007004b0, 0xc000b6e120)
	/home/prow/go/src/knative.dev/eventing/test/rekt/features/middleware_feature.go:63 +0x1c0
knative.dev/reconciler-test/pkg/feature.parallelizeStep.func1(0x2a8ff40, 0xc0007004b0, 0xc000b6e120)
	/home/prow/go/src/knative.dev/eventing/vendor/knative.dev/reconciler-test/pkg/feature/collapse.go:74 +0x79
knative.dev/reconciler-test/pkg/environment.(*MagicEnvironment).Test.func1.1(0xc000b6e120)
	/home/prow/go/src/knative.dev/eventing/vendor/knative.dev/reconciler-test/pkg/environment/magic.go:168 +0x201
testing.tRunner(0xc000b6e120, 0xc000c29bf0)
	/root/.gvm/gos/go1.14.12/src/testing/testing.go:1050 +0x1ec
created by testing.(*T).Run
	/root/.gvm/gos/go1.14.12/src/testing/testing.go:1095 +0x538

Expected behavior

No failures after the test finishes.

To Reproduce

Run TestBrokerAsMiddleware many times.

Knative release version
devel

Additional context

• [flaky] test/rekt.TestBrokerAsMiddleware/Assert #4619
• [flaky] test/rekt.TestBrokerAsMiddleware/Assert/[Stable/MUST]broker_as_middleware_deliver_an_event #4620

[lberk]

@n3wscott In case you're bored while on vacation :)

[vaikas]

Still failing.

I'll take a look

https://prow.knative.dev/view/gs/knative-prow/pr-logs/pull/knative_eventing/4727/pull-knative-eventing-integration-tests/1349302296709697536

=== RUN   TestBrokerAsMiddleware/Assert/[Stable/MUST]broker_as_middleware_deliver_an_event
=== PAUSE TestBrokerAsMiddleware/Assert/[Stable/MUST]broker_as_middleware_deliver_an_event
=== CONT  TestBrokerAsMiddleware/Assert/[Stable/MUST]broker_as_middleware_deliver_an_event
        --- FAIL: TestBrokerAsMiddleware/Assert/[Stable/MUST]broker_as_middleware_deliver_an_event (240.00s)
panic: Fail in goroutine after TestBrokerAsMiddleware/Setup has completed [recovered]
	panic: Fail in goroutine after TestBrokerAsMiddleware/Setup has completed

goroutine 255 [running]:
testing.tRunner.func1.1(0x2289cc0, 0xc000d824d0)
	/root/.gvm/gos/go1.14.12/src/testing/testing.go:999 +0x461
testing.tRunner.func1(0xc0008e8240)
	/root/.gvm/gos/go1.14.12/src/testing/testing.go:1002 +0x606
panic(0x2289cc0, 0xc000d824d0)
	/root/.gvm/gos/go1.14.12/src/runtime/panic.go:975 +0x3e3
testing.(*common).Fail(0xc0007fdc20)
	/root/.gvm/gos/go1.14.12/src/testing/testing.go:671 +0x1fc
testing.(*common).Fail(0xc0007fdd40)
	/root/.gvm/gos/go1.14.12/src/testing/testing.go:665 +0x156
testing.(*common).FailNow(0xc0007fdd40)
	/root/.gvm/gos/go1.14.12/src/testing/testing.go:693 +0x39
testing.(*common).Fatalf(0xc0007fdd40, 0x25c760d, 0x33, 0xc000844320, 0x2, 0x2)
	/root/.gvm/gos/go1.14.12/src/testing/testing.go:789 +0x9e
knative.dev/reconciler-test/pkg/eventshub.(*Store).AssertAtLeast(0xc0007976e0, 0x1, 0xc000010a68, 0x1, 0x1, 0x391ac18, 0x3e, 0xc000917288)
	/home/prow/go/src/knative.dev/eventing/vendor/knative.dev/reconciler-test/pkg/eventshub/event_info_store.go:159 +0x323
knative.dev/reconciler-test/pkg/eventshub.(*Store).AssertInRange(0xc0007976e0, 0x1, 0x1, 0xc000010a68, 0x1, 0x1, 0xc000897cd0, 0x43f548, 0x8) 	/home/prow/go/src/knative.dev/eventing/vendor/knative.dev/reconciler-test/pkg/eventshub/event_info_store.go:169 +0xb1 knative.dev/reconciler-test/pkg/eventshub.(*Store).AssertExact(0xc0007976e0, 0x1, 0xc000010a68, 0x1, 0x1, 0x3d, 0xc0004c4478, 0x3d) 	/home/prow/go/src/knative.dev/eventing/vendor/knative.dev/reconciler-test/pkg/eventshub/event_info_store.go:199 +0x99 knative.dev/eventing/test/rekt/features.BrokerAsMiddleware.func2(0x2aa97c0, 0xc000918480, 0xc0008e8240) 	/home/prow/go/src/knative.dev/eventing/test/rekt/features/middleware_feature.go:63 +0x1c0 knative.dev/reconciler-test/pkg/feature.parallelizeStep.func1(0x2aa97c0, 0xc000918480, 0xc0008e8240) 	/home/prow/go/src/knative.dev/eventing/vendor/knative.dev/reconciler-test/pkg/feature/collapse.go:74 +0x79 knative.dev/reconciler-test/pkg/environment.(*MagicEnvironment).Test.func1.1(0xc0008e8240) 	/home/prow/go/src/knative.dev/eventing/vendor/knative.dev/reconciler-test/pkg/environment/magic.go:168 +0x201 testing.tRunner(0xc0008e8240, 0xc000831260) 	/root/.gvm/gos/go1.14.12/src/testing/testing.go:1050 +0x1ec created by testing.(*T).Run 	/root/.gvm/gos/go1.14.12/src/testing/testing.go:1095 +0x538 | knative.dev/reconciler-test/pkg/eventshub.(*Store).AssertInRange(0xc0007976e0, 0x1, 0x1, 0xc000010a68, 0x1, 0x1, 0xc000897cd0, 0x43f548, 0x8) 	/home/prow/go/src/knative.dev/eventing/vendor/knative.dev/reconciler-test/pkg/eventshub/event_info_store.go:169 +0xb1 knative.dev/reconciler-test/pkg/eventshub.(*Store).AssertExact(0xc0007976e0, 0x1, 0xc000010a68, 0x1, 0x1, 0x3d, 0xc0004c4478, 0x3d) 	/home/prow/go/src/knative.dev/eventing/vendor/knative.dev/reconciler-test/pkg/eventshub/event_info_store.go:199 +0x99 knative.dev/eventing/test/rekt/features.BrokerAsMiddleware.func2(0x2aa97c0, 0xc000918480, 0xc0008e8240) 	/home/prow/go/src/knative.dev/eventing/test/rekt/features/middleware_feature.go:63 +0x1c0 knative.dev/reconciler-test/pkg/feature.parallelizeStep.func1(0x2aa97c0, 0xc000918480, 0xc0008e8240) 	/home/prow/go/src/knative.dev/eventing/vendor/knative.dev/reconciler-test/pkg/feature/collapse.go:74 +0x79 knative.dev/reconciler-test/pkg/environment.(*MagicEnvironment).Test.func1.1(0xc0008e8240) 	/home/prow/go/src/knative.dev/eventing/vendor/knative.dev/reconciler-test/pkg/environment/magic.go:168 +0x201 testing.tRunner(0xc0008e8240, 0xc000831260) 	/root/.gvm/gos/go1.14.12/src/testing/testing.go:1050 +0x1ec created by testing.(*T).Run 	/root/.gvm/gos/go1.14.12/src/testing/testing.go:1095 +0x538
-- | --
knative.dev/reconciler-test/pkg/eventshub.(*Store).AssertInRange(0xc0007976e0, 0x1, 0x1, 0xc000010a68, 0x1, 0x1, 0xc000897cd0, 0x43f548, 0x8) 	/home/prow/go/src/knative.dev/eventing/vendor/knative.dev/reconciler-test/pkg/eventshub/event_info_store.go:169 +0xb1 knative.dev/reconciler-test/pkg/eventshub.(*Store).AssertExact(0xc0007976e0, 0x1, 0xc000010a68, 0x1, 0x1, 0x3d, 0xc0004c4478, 0x3d) 	/home/prow/go/src/knative.dev/eventing/vendor/knative.dev/reconciler-test/pkg/eventshub/event_info_store.go:199 +0x99 knative.dev/eventing/test/rekt/features.BrokerAsMiddleware.func2(0x2aa97c0, 0xc000918480, 0xc0008e8240) 	/home/prow/go/src/knative.dev/eventing/test/rekt/features/middleware_feature.go:63 +0x1c0 knative.dev/reconciler-test/pkg/feature.parallelizeStep.func1(0x2aa97c0, 0xc000918480, 0xc0008e8240) 	/home/prow/go/src/knative.dev/eventing/vendor/knative.dev/reconciler-test/pkg/feature/collapse.go:74 +0x79 knative.dev/reconciler-test/pkg/environment.(*MagicEnvironment).Test.func1.1(0xc0008e8240) 	/home/prow/go/src/knative.dev/eventing/vendor/knative.dev/reconciler-test/pkg/environment/magic.go:168 +0x201 testing.tRunner(0xc0008e8240, 0xc000831260) 	/root/.gvm/gos/go1.14.12/src/testing/testing.go:1050 +0x1ec created by testing.(*T).Run 	/root/.gvm/gos/go1.14.12/src/testing/testing.go:1095 +0x538

 

[slinkydeveloper]

/assign

I'm looking at it too, 2 pair of eyes never hurt 😄

[slinkydeveloper]

Ok I may have found the problem, it's in the reconciler test execution