Skip to content

Add hacky sleep trying to mitigate the test flakiness for now#1145

Merged
knative-prow-robot merged 1 commit into
knative:masterfrom
chizhg:AddHackySleepInE2E
May 2, 2019
Merged

Add hacky sleep trying to mitigate the test flakiness for now#1145
knative-prow-robot merged 1 commit into
knative:masterfrom
chizhg:AddHackySleepInE2E

Conversation

@chizhg
Copy link
Copy Markdown
Contributor

@chizhg chizhg commented May 2, 2019

No description provided.

@googlebot googlebot added the cla: yes Indicates the PR's author has signed the CLA. label May 2, 2019
@knative-prow-robot knative-prow-robot added the size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. label May 2, 2019
@chizhg
Copy link
Copy Markdown
Contributor Author

chizhg commented May 2, 2019

/uncc @evankanderson
/uncc @n3wscott

@chizhg
Copy link
Copy Markdown
Contributor Author

chizhg commented May 2, 2019

/test pull-knative-eventing-integration-tests

2 similar comments
@chizhg
Copy link
Copy Markdown
Contributor Author

chizhg commented May 2, 2019

/test pull-knative-eventing-integration-tests

@chizhg
Copy link
Copy Markdown
Contributor Author

chizhg commented May 2, 2019

/test pull-knative-eventing-integration-tests

@chizhg
Copy link
Copy Markdown
Contributor Author

chizhg commented May 2, 2019

/cc @Harwayne
Looks it helps. 3/4 passed and the only failure was due to some cluster setup issue, which I've never met before.

@Harwayne
Copy link
Copy Markdown
Contributor

Harwayne commented May 2, 2019

/lgtm
/approve

For context, the continuous started going flaky red around da72a31 (#1119). We don't see any great reasons. The one that we looked into deeply had an error of:

 {"level":"error","ts":"2019-05-02T16:39:36.857Z","caller":"fanout/fanout_handler.go:121","msg":"Fanout had an error","error":"Unable to complete request Post http://e2e-singleevent-logger-pod-structured.test-single-structured-event-in-memory.svc.cluster.local/: dial tcp 10.43.246.111:80: i/o timeout","stacktrace":"github.com/knative/eventing/pkg/sidecar/fanout.(*Handler).dispatch\n\t/go/src/github.com/knative/eventing/pkg/sidecar/fanout/fanout_handler.go:121\ngithub.com/knative/eventing/pkg/sidecar/fanout.createReceiverFunction.func1\n\t/go/src/github.com/knative/eventing/pkg/sidecar/fanout/fanout_handler.go:99\ngithub.com/knative/eventing/pkg/provisioners.(*MessageReceiver).HandleRequest\n\t/go/src/github.com/knative/eventing/pkg/provisioners/message_receiver.go:155\ngithub.com/knative/eventing/pkg/sidecar/fanout.(*Handler).ServeHTTP\n\t/go/src/github.com/knative/eventing/pkg/sidecar/fanout/fanout_handler.go:104\ngithub.com/knative/eventing/pkg/sidecar/multichannelfanout.(*Handler).ServeHTTP\n\t/go/src/github.com/knative/eventing/pkg/sidecar/multichannelfanout/multi_channel_fanout_handler.go:105\ngithub.com/knative/eventing/pkg/sidecar/swappable.(*Handler).ServeHTTP\n\t/go/src/github.com/knative/eventing/pkg/sidecar/swappable/swappable.go:107\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2774\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:1878"}

We aren't sure of the real root cause, but the idea with this PR is to add back the time between when a pod is running and when it is expected to do its 'job'. We used to get this time for free while the istio-proxy container was setup.

This feels icky, but the number of failed test runs is too high to wait until we get a real fix.

@knative-prow-robot knative-prow-robot added the lgtm Indicates that a PR is ready to be merged. label May 2, 2019
@knative-prow-robot
Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Fredy-Z, Harwayne

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@knative-prow-robot knative-prow-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 2, 2019
@knative-prow-robot knative-prow-robot merged commit 5597d28 into knative:master May 2, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. cla: yes Indicates the PR's author has signed the CLA. lgtm Indicates that a PR is ready to be merged. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants