[flaky] test/conformance.TestBrokerV1Beta1DataPlaneConsumer/Replies_are_accepted_and_delivered

[knative-test-reporter-robot]

Auto-generated issue tracking flakiness of test

• Test name: test/conformance.TestBrokerV1Beta1DataPlaneConsumer/Replies_are_accepted_and_delivered
• Repository name: eventing

[knative-test-reporter-robot]

Latest result for this test: Passed
Last build start time: 2020-08-31 08:55:16 +0000 UTC
Failed 0 times out of 8 runs.

2020-08-31 08:55:16 +0000 UTC: ◻ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ◻ ✔
2020-08-30 08:56:40 +0000 UTC: ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔
2020-08-27 08:59:47 +0000 UTC: ✔ ✔ ✔ ✔ ✖ ✔ ✔ ✔ ✔ ✔
2020-08-26 08:53:35 +0000 UTC: ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔
2020-08-25 08:52:50 +0000 UTC: ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✖
2020-08-24 08:52:51 +0000 UTC: ✔ ✔ ✔ ✔ ✖ ◻ ✔ ✔ ✔ ✔
2020-08-23 08:53:23 +0000 UTC: ✔ ✔ ✔ ✖ ✔ ✔ ✔ ✔ ✔ ✔
2020-08-22 08:54:10 +0000 UTC: ✔ ✖ ✔ ✔ ✔ ✔ ✖ ✔ ✔ ✔
2020-08-21 08:54:46 +0000 UTC: ✔ ✔ ✖ ✔ ✖ ✔ ✔ ✔ ✖ ✖
2020-08-20 08:53:12 +0000 UTC: ✔ ✔ ✖ ✔ ✔ ✔ ✔ ✔ ✔ ✔

Older builds

2020-08-19 08:53:30 +0000 UTC: ✖ ✔ ✖ ✔ ✔ ◻ ✔ ✖ ✔ ✔
2020-08-18 08:52:48 +0000 UTC: ✔ ✔ ✖ ✔ ✔ ✔ ✔ ✖ ✔ ✔
2020-08-17 08:53:10 +0000 UTC: ✔ ✖ ✔ ✔ ✔ ✖ ✔ ✔ ✖ ✔
2020-08-16 08:54:35 +0000 UTC: ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✖ ✖ ✖
2020-08-15 08:52:44 +0000 UTC: ✔ ✖ ✔ ✔ ✖ ✔ ✔ ✔ ✔ ✖
2020-08-14 08:52:41 +0000 UTC: ✔ ✔ ✖ ✔ ✖ ✔ ✔ ✔ ✔ ✖
2020-08-13 08:53:10 +0000 UTC: ✔ ✖ ✔ ✖ ✖ ✖ ✔ ✔ ✔ ✖
2020-08-11 08:52:38 +0000 UTC: ✔ ✔ ✔ ✔ ✔ ✖ ✖ ✔ ✖ ✔
2020-08-10 08:53:55 +0000 UTC: ✔ ✔ ✔ ✖ ✔ ✔ ✔ ✔ ✔ ✔
2020-08-09 08:54:03 +0000 UTC: ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✖
2020-08-08 08:53:29 +0000 UTC: ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✖ ✖ ✔
2020-08-07 08:53:29 +0000 UTC: ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔
2020-08-06 08:55:44 +0000 UTC: ✔ ✖ ✔ ✔ ✔ ✔ ✔ ✖ ✖ ✔

✔ Passed ✖ Failed ◻ Skipped

[vaikas]

Another panic. I'm focusing on looking at this:

2020/08/20 06:12:13 http: panic serving 10.4.2.15:36126: runtime error: invalid memory address or nil pointer dereference
goroutine 729 [running]:
net/http.(*conn).serve.func1(0xc0000af400)
	net/http/server.go:1800 +0x139
panic(0x19672e0, 0x2e645c0)
	runtime/panic.go:975 +0x3e3
knative.dev/eventing/pkg/mtbroker/filter.(*Handler).sendEvent(0xc00052a900, 0x2000040, 0xc0004c88a0, 0xc0004c8180, 0xc0000924e0, 0x58, 0xc000524380, 0xc000524400, 0xc0001aca80, 0x0, ...)
	knative.dev/eventing/pkg/mtbroker/filter/filter_handler.go:258 +0x433
knative.dev/eventing/pkg/mtbroker/filter.(*Handler).send(0xc00052a900, 0x2000040, 0xc0004c88a0, 0x1ff9140, 0xc0002160a0, 0xc0004c8180, 0xc0000924e0, 0x58, 0xc000524400, 0xc000524380, ...)
	knative.dev/eventing/pkg/mtbroker/filter/filter_handler.go:210 +0xab
knative.dev/eventing/pkg/mtbroker/filter.(*Handler).ServeHTTP(0xc00052a900, 0x1ff9140, 0xc0002160a0, 0xc0002e2900)
	knative.dev/eventing/pkg/mtbroker/filter/filter_handler.go:204 +0x164f
knative.dev/eventing/pkg/health.WithReadinessCheck.func1(0x1ff9140, 0xc0002160a0, 0xc0002e2900)
	knative.dev/eventing/pkg/health/probes.go:44 +0x78
net/http.HandlerFunc.ServeHTTP(0xc0003640a0, 0x1ff9140, 0xc0002160a0, 0xc0002e2900)
	net/http/server.go:2041 +0x44
knative.dev/eventing/pkg/health.WithLivenessCheck.func1(0x1ff9140, 0xc0002160a0, 0xc0002e2900)
	knative.dev/eventing/pkg/health/probes.go:34 +0x71
net/http.HandlerFunc.ServeHTTP(0xc0003640c0, 0x1ff9140, 0xc0002160a0, 0xc0002e2900)
	net/http/server.go:2041 +0x44
go.opencensus.io/plugin/ochttp.(*Handler).ServeHTTP(0xc00035c050, 0x1ff9140, 0xc0002160a0, 0xc00065c300)
	go.opencensus.io@v0.22.4/plugin/ochttp/server.go:92 +0x27e
net/http.serverHandler.ServeHTTP(0xc000370000, 0x1ff7ac0, 0xc0002ec000, 0xc00065c300)
	net/http/server.go:2836 +0xa3
net/http.(*conn).serve(0xc0000af400, 0x1ffff80, 0xc000366000)
	net/http/server.go:1924 +0x86c
created by net/http.(*Server).Serve
	net/http/server.go:2962 +0x35c

[vaikas]

After fixing the panic. I now see these failures instead of panics. Digging into those next...

{"level":"error","ts":"2020-08-20T19:08:35.690Z","logger":"mt_broker_filter","caller":"filter/filter_handler.go:212","msg":"failed to send event","commit":"a6db2a0","error":"failed to dispatch message: Post \"http://transformer-pod.test-broker-v1-beta1-data-plane-consumer-fl8wb.svc.cluster.local/\": dial tcp 10.12.8.23:80: connect: connection refused","stacktrace":"knative.dev/eventing/pkg/mtbroker/filter.(*Handler).send\n\tknative.dev/eventing/pkg/mtbroker/filter/filter_handler.go:212\nknative.dev/eventing/pkg/mtbroker/filter.(*Handler).ServeHTTP\n\tknative.dev/eventing/pkg/mtbroker/filter/filter_handler.go:204\nknative.dev/eventing/pkg/health.WithReadinessCheck.func1\n\tknative.dev/eventing/pkg/health/probes.go:44\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2012\nknative.dev/eventing/pkg/health.WithLivenessCheck.func1\n\tknative.dev/eventing/pkg/health/probes.go:34\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2012\ngo.opencensus.io/plugin/ochttp.(*Handler).ServeHTTP\n\tgo.opencensus.io@v0.22.4/plugin/ochttp/server.go:92\nnet/http.serverHandler.ServeHTTP\n\tnet/http/server.go:2807\nnet/http.(*conn).serve\n\tnet/http/server.go:1895"}

[vaikas]

My next hunch/guess/whatevs is that we make sure pod is up, but the service is not up yet or endpoints are not there, and hence it's actually not ready. Looking at this next...

[vaikas]

all the failed tests are still connection refused:

mt-broker-filter-64999b6b85-lfhmt-filter.log:{"level":"error","ts":"2020-08-24T12:20:54.589Z","logger":"mt_broker_filter","caller":"filter/filter_handler.go:212","msg":"failed to send event","commit":"ae90af3","error":"failed to dispatch message: Post \"http://transformer-pod.test-broker-v1-beta1-data-plane-consumer-t8qfm.svc.cluster.local/\": dial tcp 10.19.246.46:80: connect: connection refused","stacktrace":"knative.dev/eventing/pkg/mtbroker/filter.(*Handler).send\n\tknative.dev/eventing/pkg/mtbroker/filter/filter_handler.go:212\nknative.dev/eventing/pkg/mtbroker/filter.(*Handler).ServeHTTP\n\tknative.dev/eventing/pkg/mtbroker/filter/filter_handler.go:204\nknative.dev/eventing/pkg/health.WithReadinessCheck.func1\n\tknative.dev/eventing/pkg/health/probes.go:44\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2041\nknative.dev/eventing/pkg/health.WithLivenessCheck.func1\n\tknative.dev/eventing/pkg/health/probes.go:34\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2041\ngo.opencensus.io/plugin/ochttp.(*Handler).ServeHTTP\n\tgo.opencensus.io@v0.22.4/plugin/ochttp/server.go:92\nnet/http.serverHandler.ServeHTTP\n\tnet/http/server.go:2836\nnet/http.(*conn).serve\n\tnet/http/server.go:1924"}