Verifying Eventing samples for 0.3 release

[samodell]

• GCP pubsub - @mogar1980
• Github source - @bbrowning
• Iot core - @grantr
• K8s - @csantanapr

[bbrowning]

It looks like eventing and eventing-sources 0.3 were released - I'll get the GitHub source verified by the end of today.

[bbrowning]

I can't check the actual checkbox, but I did a fresh install of Knative 0.3 on GKE, followed the new 0.3 instructions for setting up a static ip and custom domain (as linked from the GitHub source sample), and was able to verify the GitHub source sample still works without any changes.

[samodell]

Thanks @bbrowning ! Updated.

[csantanapr]

There are problems with the k8s sample
I tried 0.3 eventing with minikube 0.32.0 and k8s 1.13.2
It doesn't work, message-dumper never runs
From the logs I see this for the testchannel

$ kubectl logs controller-manager-0 -n knative-sources
2019/01/11 05:49:50 Registering Components.
2019/01/11 05:49:50 Skipping the AWS SQS Source controller.
2019/01/11 05:49:50 Adding the Cron Job Source controller.
2019/01/11 05:49:50 Starting the Cmd.
{"level":"info","ts":1547185888.8160148,"logger":"fallback","caller":"sdk/reconciler.go:52","msg":"Reconciling &TypeMeta{Kind:,APIVersion:,} default/testevents-qh2vp"}
{"level":"warn","ts":1547185888.9313293,"logger":"fallback","caller":"sdk/reconciler.go:74","msg":"Failed to reconcile &TypeMeta{Kind:,APIVersion:,}: sink \"default/testchannel\" (eventing.knative.dev/v1alpha1, Kind=Channel) does not contain address"}

Logs from controller

$ kubectl logs eventing-controller-cfbb757bd-kmg7k -n knative-eventing
{"level":"info","ts":"2019-01-11T05:49:42.572Z","caller":"logging/config.go:96","msg":"Successfully created the logger.","knative.dev/jsonconfig":"{\n  \"level\": \"info\",\n  \"development\": false,\n  \"outputPaths\": [\"stdout\"],\n  \"errorOutputPaths\": [\"stderr\"],\n  \"encoding\": \"json\",\n  \"encoderConfig\": {\n    \"timeKey\": \"ts\",\n    \"levelKey\": \"level\",\n    \"nameKey\": \"logger\",\n    \"callerKey\": \"caller\",\n    \"messageKey\": \"msg\",\n    \"stacktraceKey\": \"stacktrace\",\n    \"lineEnding\": \"\",\n    \"levelEncoder\": \"\",\n    \"timeEncoder\": \"iso8601\",\n    \"durationEncoder\": \"\",\n    \"callerEncoder\": \"\"\n  }\n}\n"}
{"level":"info","ts":"2019-01-11T05:49:42.572Z","caller":"logging/config.go:97","msg":"Logging level set to info"}
{"level":"info","ts":"2019-01-11T05:49:42.572Z","logger":"controller","caller":"controller/main.go:66","msg":"Starting the Controller","commit":"4712e3a","knative.dev/controller":"controller"}
{"level":"info","ts":"2019-01-11T05:49:42.678Z","logger":"controller","caller":"controller/main.go:99","msg":"Starting metrics listener at %s:9090","commit":"4712e3a","knative.dev/controller":"controller"}
{"level":"info","ts":1547185782.754786,"logger":"kubebuilder.controller","caller":"controller/controller.go:120","msg":"Starting EventSource","Controller":"subscription-controller","Source":{"Type":{"metadata":{"creationTimestamp":null},"spec":{"channel":{}},"status":{"physicalSubscription":{}}}}}
{"level":"info","ts":1547185782.861732,"logger":"kubebuilder.controller","caller":"controller/controller.go:134","msg":"Starting Controller","Controller":"subscription-controller"}
{"level":"info","ts":1547185782.9624798,"logger":"kubebuilder.controller","caller":"controller/controller.go:153","msg":"Starting workers","Controller":"subscription-controller","WorkerCount":1}
I0111 05:51:34.922540       1 reconcile.go:51] Reconciling subscription default/testevents-subscription
I0111 05:51:34.922634       1 reconcile.go:87] DeletionTimestamp: <nil>
W0111 05:51:34.951357       1 reconcile.go:230] Failed to fetch SubscriberSpec target &ObjectReference{Kind:Service,Namespace:,Name:message-dumper,UID:,APIVersion:serving.knative.dev/v1alpha1,ResourceVersion:,FieldPath:,}: services.serving.knative.dev "message-dumper" not found
W0111 05:51:34.951443       1 reconcile.go:114] Failed to resolve Subscriber {Ref:&ObjectReference{Kind:Service,Namespace:,Name:message-dumper,UID:,APIVersion:serving.knative.dev/v1alpha1,ResourceVersion:,FieldPath:,} DNSName:<nil>} : services.serving.knative.dev "message-dumper" not found
{"level":"error","ts":1547185894.9849646,"logger":"kubebuilder.controller","caller":"controller/controller.go:209","msg":"Reconciler error","Controller":"subscription-controller","Request":"default/testevents-subscription","error":"services.serving.knative.dev \"message-dumper\" not found","stacktrace":"github.com/knative/eventing/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/argent/go/src/github.com/knative/eventing/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/knative/eventing/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/argent/go/src/github.com/knative/eventing/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:209\ngithub.com/knative/eventing/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/home/argent/go/src/github.com/knative/eventing/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:157\ngithub.com/knative/eventing/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/home/argent/go/src/github.com/knative/eventing/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/knative/eventing/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/argent/go/src/github.com/knative/eventing/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/knative/eventing/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/home/argent/go/src/github.com/knative/eventing/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
I0111 05:51:35.985540       1 reconcile.go:51] Reconciling subscription default/testevents-subscription
I0111 05:51:35.985938       1 reconcile.go:87] DeletionTimestamp: <nil>
W0111 05:51:36.004782       1 reconcile.go:114] Failed to resolve Subscriber {Ref:&ObjectReference{Kind:Service,Namespace:,Name:message-dumper,UID:,APIVersion:serving.knative.dev/v1alpha1,ResourceVersion:,FieldPath:,} DNSName:<nil>} : status does not contain address
{"level":"error","ts":1547185896.0050697,"logger":"kubebuilder.controller","caller":"controller/controller.go:209","msg":"Reconciler error","Controller":"subscription-controller","Request":"default/testevents-subscription","error":"status does not contain address","stacktrace":"github.com/knative/eventing/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/argent/go/src/github.com/knative/eventing/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/knative/eventing/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/argent/go/src/github.com/knative/eventing/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:209\ngithub.com/knative/eventing/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/home/argent/go/src/github.com/knative/eventing/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:157\ngithub.com/knative/eventing/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/home/argent/go/src/github.com/knative/eventing/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/knative/eventing/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/argent/go/src/github.com/knative/eventing/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/knative/eventing/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/home/argent/go/src/github.com/knative/eventing/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
I0111 05:51:37.007486       1 reconcile.go:51] Reconciling subscription default/testevents-subscription

[srvaroa]

I am working on moving the AWS SQS source to eventing-sources/contrib (see knative/eventing-contrib#177) and found the same issue yesterday. I was assuming it'd be something I had broken during the refactor but I may be hitting the same issue.

[csantanapr]

I tried again with different version of k8s, now using minikube with k8s 1.11.6 and get the same errors

kubectl logs controller-manager-0 -n knative-sources
{"level":"info","ts":1547218967.8800313,"logger":"fallback","caller":"sdk/reconciler.go:52","msg":"Reconciling &TypeMeta{Kind:,APIVersion:,} default/testevents-qx7cn"}
{"level":"warn","ts":1547218967.8907819,"logger":"fallback","caller":"sdk/reconciler.go:74","msg":"Failed to reconcile &TypeMeta{Kind:,APIVersion:,}: sink \"default/testchannel\" (eventing.knative.dev/v1alpha1, Kind=Channel) does not contain address"}

in the the eventing controller

kubectl logs eventing-controller-847d8cf969-dzwzq -n knative-eventing
{"level":"info","ts":1547218501.9501147,"logger":"kubebuilder.controller","caller":"controller/controller.go:153","msg":"Starting workers","Controller":"subscription-controller","WorkerCount":1}
I0111 15:02:57.075128       1 reconcile.go:51] Reconciling subscription default/testevents-subscription
I0111 15:02:57.075358       1 reconcile.go:87] DeletionTimestamp: <nil>
W0111 15:02:57.089030       1 reconcile.go:230] Failed to fetch SubscriberSpec target &ObjectReference{Kind:Service,Namespace:,Name:message-dumper,UID:,APIVersion:serving.knative.dev/v1alpha1,ResourceVersion:,FieldPath:,}: services.serving.knative.dev "message-dumper" not found
W0111 15:02:57.089100       1 reconcile.go:114] Failed to resolve Subscriber {Ref:&ObjectReference{Kind:Service,Namespace:,Name:message-dumper,UID:,APIVersion:serving.knative.dev/v1alpha1,ResourceVersion:,FieldPath:,} DNSName:<nil>} : services.serving.knative.dev "message-dumper" not found
{"level":"error","ts":1547218977.0960505,"logger":"kubebuilder.controller","caller":"controller/controller.go:209","msg":"Reconciler error","Controller":"subscription-controller","Request":"default/testevents-subscription","error":"services.serving.knative.dev \"message-dumper\" not found","stacktrace":"github.com/knative/eventing/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/argent/go/src/github.com/knative/eventing/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/knative/eventing/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/argent/go/src/github.com/knative/eventing/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:209\ngithub.com/knative/eventing/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/home/argent/go/src/github.com/knative/eventing/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:157\ngithub.com/knative/eventing/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/home/argent/go/src/github.com/knative/eventing/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/knative/eventing/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/argent/go/src/github.com/knative/eventing/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/knative/eventing/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/home/argent/go/src/github.com/knative/eventing/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
I0111 15:02:58.096549       1 reconcile.go:51] Reconciling subscription default/testevents-subscription
I0111 15:02:58.097026       1 reconcile.go:87] DeletionTimestamp: <nil>
W0111 15:02:58.101593       1 reconcile.go:114] Failed to resolve Subscriber {Ref:&ObjectReference{Kind:Service,Namespace:,Name:message-dumper,UID:,APIVersion:serving.knative.dev/v1alpha1,ResourceVersion:,FieldPath:,} DNSName:<nil>} : status does not contain address
{"level":"error","ts":1547218978.1018186,"logger":"kubebuilder.controller","caller":"controller/controller.go:209","msg":"Reconciler error","Controller":"subscription-controller","Request":"default/testevents-subscription","error":"status does not contain address","stacktrace":"github.com/knative/eventing/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/argent/go/src/github.com/knative/eventing/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/knative/eventing/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/argent/go/src/github.com/knative/eventing/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:209\ngithub.com/knative/eventing/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/home/argent/go/src/github.com/knative/eventing/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:157\ngithub.com/knative/eventing/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/home/argent/go/src/github.com/knative/eventing/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/knative/eventing/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/argent/go/src/github.com/knative/eventing/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/knative/eventing/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/home/argent/go/src/github.com/knative/eventing/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}

[srvaroa]

Reviewing the latest commits I noticed that this introduces some empty status objects in the channel: knative/eventing@6b891e9 - I haven't been able to confirm yet if it's related but just in case it gives a useful clue.

[grantr]

@srvaroa if you're talking about this:

+  subresources:
+    status: {}

That's the correct way to enable the status subresource on the CRD. It could still be the source of the issue, but it's not a typo. 😄

[srvaroa]

Yes! What I meant is that after adding the empty status, the subscriber reconcile loop may not be handling the object properly (as the error complains for lack of Address inside the status).

[srvaroa]

I can confirm from my side that the k8s sample doesn't work, same error as here: (#733 (comment)). This is a fresh install of v0.3.0 on GKE.

After updating to knative/eventing@5dfd2c8 and knative/eventing-contrib@6fde0bf it does seem to work well following the README. The same revisions also resolve the same problem I was hitting at knative/eventing-contrib#177.