TestChannelTracing fails

[vaikas]

Describe the bug

TestChannelTracing fails:
https://github.com/mattmoor/mink/runs/1701843508?check_suite_focus=true#step:11:43

        tracing_test_helper.go:90: Unable to get trace "fd12d2a6718a81661bf6913994a75c5b": timeout getting JSONTrace, most recent error:<nil>. Trace so far [{"timestamp":1610630045699653,"duration":136,"traceId":"fd12d2a6718a81661bf6913994a75c5b","id":"f29562d89101d91b","parentId":"4b64567e35cb1d16","name":"/","kind":"SERVER","localEndpoint":{"serviceName":"imc-dispatcher","ipv4":"10.96.233.120"},"annotations":[{"timestamp":1610630045699664,"value":"Received 65 bytes"}],"tags":{"http.host":"ch-kn-channel.test-channel-tracing-with-reply-in-memory-channel-messagin2dc96.svc.c485363944.local","http.method":"POST","http.path":"/","http.status_code":"202","http.url":"/","http.user_agent":"Go-http-client/1.1","opencensus.status_description":"OK"}},{"timestamp":1610630045699881,"duration":55401,"traceId":"fd12d2a6718a81661bf6913994a75c5b","id":"ef69bb4d66240005","parentId":"f29562d89101d91b","name":"knative.dev","kind":"CLIENT","localEndpoint":{"serviceName":"imc-dispatcher","ipv4":"10.96.233.120"},"tags":{"cloudevents.id":"d265b903-cf06-496e-b393-9edd34e97946","cloudevents.source":"sender","cloudevents.specversion":"1.0","cloudevents.type":"dev.knative.test.event"}},{"timestamp":1610630045699910,"duration":93412,"traceId":"fd12d2a6718a81661bf6913994a75c5b","id":"ec3d14c33a4727ee","parentId":"ef69bb4d66240005","name":"/","kind":"CLIENT","localEndpoint":{"serviceName":"imc-dispatcher","ipv4":"10.96.233.120"},"tags":{"http.host":"mutator.test-channel-tracing-with-reply-in-memory-channel-messagin2dc96.svc.c485363944.local","http.method":"POST","http.path":"/","http.status_code":"200","http.url":"http://mutator.test-channel-tracing-with-reply-in-memory-channel-messagin2dc96.svc.c485363944.local/","opencensus.status_description":"OK"}},{"timestamp":1610630045793415,"duration":322,"traceId":"fd12d2a6718a81661bf6913994a75c5b","id":"e3b91e23b8af9ca9","parentId":"e6e5c5ade38c75c0","name":"/","kind":"SERVER","localEndpoint":{"serviceName":"imc-dispatcher","ipv4":"10.96.233.120"},"tags":{"http.host":"reply-ch-kn-channel.test-channel-tracing-with-reply-in-memory-channel-messagin2dc96.svc.c485363944.local","http.method":"POST","http.path":"/","http.status_code":"202","http.url":"/","http.user_agent":"Go-http-client/1.1","opencensus.status_description":"OK"}},{"timestamp":1610630045755526,"duration":38713,"traceId":"fd12d2a6718a81661bf6913994a75c5b","id":"e6e5c5ade38c75c0","parentId":"e9116d380f6a4ed7","kind":"CLIENT","localEndpoint":{"serviceName":"imc-dispatcher","ipv4":"10.96.233.120"},"tags":{"http.host":"reply-ch-kn-channel.test-channel-tracing-with-reply-in-memory-channel-messagin2dc96.svc.c485363944.local","http.method":"POST","http.path":"","http.status_code":"202","http.url":"http://reply-ch-kn-channel.test-channel-tracing-with-reply-in-memory-channel-messagin2dc96.svc.c485363944.local","opencensus.status_description":"OK"}},{"timestamp":1610630045755446,"duration":38923,"traceId":"fd12d2a6718a81661bf6913994a75c5b","id":"e9116d380f6a4ed7","parentId":"ef69bb4d66240005","name":"knative.dev","kind":"CLIENT","localEndpoint":{"serviceName":"imc-dispatcher","ipv4":"10.96.233.120"},"tags":{"cloudevents.id":"d265b903-cf06-496e-b393-9edd34e97946","cloudevents.source":"sender","cloudevents.specversion":"1.0","cloudevents.type":"mutated"}},{"timestamp":1610630045793937,"duration":46548,"traceId":"fd12d2a6718a81661bf6913994a75c5b","id":"dd61d00d61f5ea7b","parentId":"e08d77988cd2c392","name":"/","kind":"CLIENT","localEndpoint":{"serviceName":"imc-dispatcher","ipv4":"10.96.233.120"},"tags":{"http.host":"recordevents.test-channel-tracing-with-reply-in-memory-channel-messagin2dc96.svc.c485363944.local","http.method":"POST","http.path":"/","http.status_code":"202","http.url":"http://recordevents.test-channel-tracing-with-reply-in-memory-channel-messagin2dc96.svc.c485363944.local/","opencensus.status_description":"OK"}},{"timestamp":1610630045793904,"duration":46780,"traceId":"fd12d2a6718a81661bf6913994a75c5b","id":"e08d77988cd2c392","parentId":"e3b91e23b8af9ca9","name":"knative.dev","kind":"CLIENT","localEndpoint":{"serviceName":"imc-dispatcher","ipv4":"10.96.233.120"},"tags":{"cloudevents.id":"d265b903-cf06-496e-b393-9edd34e97946","cloudevents.source":"sender","cloudevents.specversion":"1.0","cloudevents.type":"mutated"}},{"timestamp":1610630045832785,"duration":7487,"traceId":"fd12d2a6718a81661bf6913994a75c5b","id":"18d0948bdad6c310","parentId":"dd61d00d61f5ea7b","name":"/","kind":"SERVER","localEndpoint":{"serviceName":"recordevents","ipv4":"10.244.2.17"},"annotations":[{"timestamp":1610630045832796,"value":"Received 65 bytes"}],"tags":{"http.host":"recordevents.test-channel-tracing-with-reply-in-memory-channel-messagin2dc96.svc.c485363944.local","http.method":"POST","http.path":"/","http.status_code":"202","http.url":"/","http.user_agent":"Go-http-client/1.1","opencensus.status_description":"OK"}},{"timestamp":1610630045746494,"duration":8372,"traceId":"fd12d2a6718a81661bf6913994a75c5b","id":"2c8f8da6390410c2","parentId":"ec3d14c33a4727ee","name":"/","kind":"SERVER","localEndpoint":{"serviceName":"mutator","ipv4":"10.244.3.18"},"annotations":[{"timestamp":1610630045746510,"value":"Received 65 bytes"},{"timestamp":1610630045754854,"value":"Sent 65 bytes"}],"tags":{"http.host":"mutator.test-channel-tracing-with-reply-in-memory-channel-messagin2dc96.svc.c485363944.local","http.method":"POST","http.path":"/","http.status_code":"200","http.url":"/","http.user_agent":"Go-http-client/1.1","opencensus.status_description":"OK"}},{"timestamp":1610630045644896,"duration":55423,"traceId":"fd12d2a6718a81661bf6913994a75c5b","id":"4b64567e35cb1d16","parentId":"04e326563dc03233","kind":"CLIENT","localEndpoint":{"serviceName":"sender","ipv4":"10.244.2.21"},"tags":{"http.host":"","http.method":"POST","http.path":"","http.status_code":"202","http.url":"http://ch-kn-channel.test-channel-tracing-with-reply-in-memory-channel-messagin2dc96.svc.c485363944.local","opencensus.status_description":"OK"}},{"timestamp":1610630045644841,"duration":55495,"traceId":"fd12d2a6718a81661bf6913994a75c5b","id":"04e326563dc03233","name":"cloudevents.client","kind":"CLIENT","localEndpoint":{"serviceName":"sender","ipv4":"10.244.2.21"},"tags":{"cloudevents.datacontenttype":"application/json","cloudevents.id":"d265b903-cf06-496e-b393-9edd34e97946","cloudevents.source":"sender","cloudevents.specversion":"1.0","cloudevents.type":"dev.knative.test.event"}}]

Expected behavior
Tests to pass

To Reproduce
First time I've seen this

Knative release version
HEAD

Additional context
Add any other context about the problem here such as proposed priority

[vaikas]

@xtreme-sameer-vohra would you mind taking a look since you were recently working with the traces?

[mattmoor]

I wonder if mink just needs to pull this in 🤔

[mattmoor]

#4659

[mattmoor]

Should land in this morning's automated PR: https://github.com/knative-sandbox/knobots/runs/1702179030?check_suite_focus=true

[xtreme-sameer-vohra]

Looking at the failure, @mattmoor is right that the change doesn't appear to be propagated.

Ref doesn't include the change in #4659

[xtreme-sameer-vohra]

Is this failing because Github Actions uses the HEAD for the test definitions and not the vendor'ed version of eventing ?

[mattmoor]

That's what I'm thinking. Things passed in the latest update PR, so let's close until we see further issues.