Run the chaosduck component during our e2e testing.

[mattmoor]

No description provided.

[mattmoor]

It passed, running it again.

/test pull-knative-eventing-integration-tests

[mattmoor]

Pretty sure this isn't running properly yet. Without a replicated webhook, I'd at least expect intermittent failures there, and looking through the logs the pods are far too old relative to the chaos duck:

chaosduck-5989f5cd88-mrb4n             1/1   Running   0     103s
eventing-controller-76cbd5d948-gcvbk   1/1   Running   0     2m38s
eventing-webhook-b66887bcd-ctnk6       1/1   Running   1     2m32s
eventing-webhook-b66887bcd-hr8x2       1/1   Running   0     2m32s
imc-controller-6c7979cfd9-ft7kz        1/1   Running   0     4s
imc-dispatcher-6596db8fcc-wk82v        1/1   Running   0     4s

[mattmoor]

Cracking the logs I see:

knative-eventing-9utkcux231/chaosduck-5989f5cd88-mrb4n[chaosduck]: 2020/07/12 15:00:22 Ended iteration with err: pods "eventing-controller-76cbd5d948-gcvbk" is forbidden: User "system:serviceaccount:knative-eventing-9utkcux231:eventing-controller" cannot delete resource "pods" in API group "" in the namespace "knative-eventing-9utkcux231"

I just need to be less lazy about piggybacking on an existing SA and create my own with the right RBAC 🙃

[mattmoor]

This should be fixed, let's see what breaks now 😈

[mattmoor]

The problem now is where I'm standing things up. I saw this in serving where the wait_for_ready_pods in the serving namespace never completes because there is always a pod in terminating 😈

[mattmoor]

Enabling the mt-broker-controller and some Brokers start failing to become ready 🤔

Rolled back the broker bit and trying the next component on my list.

[mattmoor]

/hold

[mattmoor]

/test pull-knative-eventing-integration-tests

(running again to flush out flakes) 😇

[mattmoor]

One more time...

/test pull-knative-eventing-integration-tests

[mattmoor]

This looks like the webhook shutdown failure I am chasing (here):

TestChannelNamespaceDefaulter/InMemoryChannel-messaging.knative.dev/v1: creation.go:79: Failed to create channel "e2e-defaulter-channel": Internal error occurred: failed calling webhook "webhook.eventing.knative.dev": Post https://eventing-webhook.knative-eventing-cjvz5x2e0p.svc:443/defaulting?timeout=2s: EOF

/retest

[knative-test-reporter-robot]

The following jobs failed:

Test name	Triggers	Retries
pull-knative-eventing-integration-tests		0/3

Failed non-flaky tests preventing automatic retry of pull-knative-eventing-integration-tests:

test/e2e.TestDefaultBrokerWithManyTriggers
test/e2e.TestDefaultBrokerWithManyTriggers/test_default_broker_with_many_attribute_and_extension_triggers

[mattmoor]

Webhook again:

TestDefaultBrokerWithManyTriggers/test_default_broker_with_many_attribute_and_extension_triggers: creation.go:219: Failed to create v1beta1 trigger "trigger-testany-testany--extname1-extval1": Internal error occurred: failed calling webhook "validation.webhook.eventing.knative.dev": Post https://eventing-webhook.knative-eventing-kckqbzjwly.svc:443/resource-validation?timeout=2s: EOF

There is half a fix in (#3596), and I talked to @tcnghia about bumping network.DefaultDrainTimeout as well.

/retest

[vaikas]

/lgtm
/approve

[knative-prow-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mattmoor, vaikas

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [mattmoor,vaikas]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[mattmoor]

/hold cancel

If we start seeing pervasive issues, we should role this back, but the scope of this is intended to be a relatively stable subset to start flushing out more niche HA issues.