Make Knative eventing channels more serverless and scalable

[aslom]

Problem
Today channels do not scale in response to number of events: if channel is not used it occupies resources (memory, cpu) and may stop processing events (if hitting memory, cpu limits)

Persona:
Which persona is this feature for?

System Integrator, Event consumer (developer)

Exit Criteria

In-memory channel and one other channel (such as Kafka) can scale down and up

Time Estimate (optional):
How many developer-days do you think this may take to resolve?

weeks

Additional context (optional)
Add any other context about the feature request here.

Ref #2152

[Harwayne]

We have discussed using Knative Services in particular in the Broker Deployments. Knative Services would allow it to scale down to zero and up as needed (which would probably help Broker performance too). Our hesitation is that currently, Knative Eventing does not require Knative Serving. Switching to Knative Services would make it a hard dependency.

I think the first step would be to create a PodSpecable informer-lister. That would allow us to run our data plane pieces as either Deployments or Knative Services, without having a run time dependency on Knative Serving. The controllers would choose which to use via a ConfigMap.

This is the only reference I can find to PodSpecable, but that is a test file.

[aslom]

I really like idea to have it configurable duck-typed for channel (and sources?) and avoiding hard dependency on ksvc (keeping things simple by default)

Eventually all channel implementations could share then the same basic code so they can Deployments or Knative Services and be scalable :)

[nachocano]

I really like idea to have it configurable duck-typed for channel (and sources?) and avoiding hard dependency on ksvc (keeping things simple by default)

Eventually all channel implementations could share then the same basic code so they can Deployments or Knative Services and be scalable :)

Some initial work I did on broker ingress and filter deployments to make them "configurable": either Deployment or ksvc. Very initial work, but I guess whatever we decide for channel/sources, we can do the same for brokers.
The ideal thing as @Harwayne mentioned would be not to add a strong dependency to serving and have some sort of dynamic tracker.

[aslom]

@nachocano @Harwayne what do you mean exactly by strong dependency? Does it mean that code is not importing libraries from serving?

[Harwayne]

@nachocano @Harwayne what do you mean exactly by strong dependency? Does it mean that code is not importing libraries from serving?

The controller must run correctly if it is set to use Deployments and the Knative Serving CRDs are not installed.

Using a straightforward, auto-generated, informer-lister for ksvcs, it would fail loop saying that it cannot list that kind int he given API version.

Ideally, we would also not have a compile time dependency, but I am more flexible on that.

[lionelvillard]

I do have a dynamic informer: https://github.com/lionelvillard/knative-functions-controller/tree/master/pkg/dynamic

Maybe I can move it to pkg if not there yet.

[lionelvillard]

Full disclosure, the original code is there: https://github.com/kubernetes/client-go/tree/master/dynamic

I just made it injection compatible and at the time I used an old client go

[aslom]

After reviewing with @lionelvillard it seems that the easiest first step is to modify eventing/config/channels/in-memory-channel/500-dispatcher.yaml
to make it into ksvc and see dispatcher can be started? @Harwayne @nachocano

That requires to disable checks that assume deployment (reconcile in eventing/pkg/reconciler/inmemorychannel/controller/inmemorychannel.go).

I am going to open WIP PR to have it for testing and review.

If that works then next step is to use ConfigMap to configure dispatcher to use deployment or ksvc - that will change how "static" is dispatcher.

[aslom]

Update on experimenting - after converting imc-dispatcher into ksvc seems to run into race condition between two containers inside that try to bind to 9090 (see below).

Going to investigate more and see if I can get disable for now binding to 9090 to get imc dispatcher started and be able to dispatch one event.

(base) aslom@m eventing (channels-ksvc) $ k -n knative-eventing logs imc-dispatcher-wmjxg-deployment-8458d9b9cb-gq95d -c queue-proxy
{"level":"info","ts":"2019-11-08T19:20:14.593Z","caller":"logging/config.go:100","msg":"Successfully created the logger.","knative.dev/jsonconfig":"{\n  \"level\": \"info\",\n  \"development\": false,\n  \"outputPaths\": [\"stdout\"],\n  \"errorOutputPaths\": [\"stderr\"],\n  \"encoding\": \"json\",\n  \"encoderConfig\": {\n    \"timeKey\": \"ts\",\n    \"levelKey\": \"level\",\n    \"nameKey\": \"logger\",\n    \"callerKey\": \"caller\",\n    \"messageKey\": \"msg\",\n    \"stacktraceKey\": \"stacktrace\",\n    \"lineEnding\": \"\",\n    \"levelEncoder\": \"\",\n    \"timeEncoder\": \"iso8601\",\n    \"durationEncoder\": \"\",\n    \"callerEncoder\": \"\"\n  }\n}"}
{"level":"info","ts":"2019-11-08T19:20:14.593Z","caller":"logging/config.go:101","msg":"Logging level set to info"}
{"level":"warn","ts":"2019-11-08T19:20:14.593Z","caller":"logging/config.go:69","msg":"Fetch GitHub commit ID from kodata failed: open /var/run/ko/HEAD: permission denied"}
{"level":"error","ts":"2019-11-08T19:20:14.594Z","logger":"queueproxy","caller":"queue/main.go:448","msg":"Failed to bring up queue-proxy, shutting down.","knative.dev/key":"knative-eventing/imc-dispatcher-wmjxg","knative.dev/pod":"imc-dispatcher-wmjxg-deployment-8458d9b9cb-gq95d","error":"metrics server failed: listen tcp :9090: bind: address already in use","errorVerbose":"listen tcp :9090: bind: address already in use\nmetrics server failed\nmain.main.func3\n\t/home/prow/go/src/knative.dev/serving/cmd/queue/main.go:423\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1337","stacktrace":"main.main\n\t/home/prow/go/src/knative.dev/serving/cmd/queue/main.go:448\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:200"}

(base) aslom@m eventing (channels-ksvc) $ k -n knative-eventing logs imc-dispatcher-wmjxg-deployment-8458d9b9cb-gq95d -c user-container
2019/11/08 19:18:43 Registering 3 clients
2019/11/08 19:18:43 Registering 2 informer factories
2019/11/08 19:18:43 Registering 1 informers
2019/11/08 19:18:43 Registering 1 controllers
{"level":"info","ts":"2019-11-08T19:18:44.052Z","caller":"logging/config.go:108","msg":"Successfully created the logger.","knative.dev/jsonconfig":"{\n  \"level\": \"info\",\n  \"development\": false,\n  \"outputPaths\": [\"stdout\"],\n  \"errorOutputPaths\": [\"stderr\"],\n  \"encoding\": \"json\",\n  \"encoderConfig\": {\n    \"timeKey\": \"ts\",\n    \"levelKey\": \"level\",\n    \"nameKey\": \"logger\",\n    \"callerKey\": \"caller\",\n    \"messageKey\": \"msg\",\n    \"stacktraceKey\": \"stacktrace\",\n    \"lineEnding\": \"\",\n    \"levelEncoder\": \"\",\n    \"timeEncoder\": \"iso8601\",\n    \"durationEncoder\": \"\",\n    \"callerEncoder\": \"\"\n  }\n}\n"}
{"level":"info","ts":"2019-11-08T19:18:44.052Z","caller":"logging/config.go:109","msg":"Logging level set to info"}
{"level":"info","ts":"2019-11-08T19:18:44.061Z","logger":"inmemorychannel_dispatcher.in-memory-channel-dispatcher","caller":"dispatcher/controller.go:86","msg":"Setting up event handlers","commit":"e79eab7","knative.dev/controller":"in-memory-channel-dispatcher"}
{"level":"info","ts":"2019-11-08T19:18:44.062Z","logger":"inmemorychannel_dispatcher","caller":"profiling/server.go:60","msg":"Profiling enabled: false","commit":"e79eab7"}
{"level":"info","ts":"2019-11-08T19:18:44.084Z","logger":"inmemorychannel_dispatcher","caller":"metrics/exporter.go:94","msg":"Flushing the existing exporter before setting up the new exporter.","commit":"e79eab7"}
{"level":"info","ts":"2019-11-08T19:18:44.084Z","logger":"inmemorychannel_dispatcher","caller":"metrics/prometheus_exporter.go:37","msg":"Created Opencensus Prometheus exporter with config: &{knative.dev/inmemorychannel-dispatcher inmemorychannel_dispatcher prometheus 5000000000 <nil> 9090 false   {   false}}. Start the server for Prometheus exporter.","commit":"e79eab7"}
{"level":"info","ts":"2019-11-08T19:18:44.084Z","logger":"inmemorychannel_dispatcher","caller":"metrics/exporter.go:103","msg":"Successfully updated the metrics exporter; old config: <nil>; new config &{knative.dev/inmemorychannel-dispatcher inmemorychannel_dispatcher prometheus 5000000000 <nil> 9090 false   {   false}}","commit":"e79eab7"}
{"level":"info","ts":"2019-11-08T19:18:44.084Z","logger":"inmemorychannel_dispatcher.in-memory-channel-dispatcher","caller":"configmap/store.go:157","msg":"tracing-config config \"config-tracing\" config was added or updated: &config.Config{Backend:\"none\", ZipkinEndpoint:\"\", StackdriverProjectID:\"\", Debug:false, SampleRate:0.1}","commit":"e79eab7","knative.dev/controller":"in-memory-channel-dispatcher"}
{"level":"info","ts":"2019-11-08T19:18:44.162Z","logger":"inmemorychannel_dispatcher","caller":"sharedmain/main.go:179","msg":"Starting informers.","commit":"e79eab7"}
{"level":"info","ts":"2019-11-08T19:18:44.262Z","logger":"inmemorychannel_dispatcher","caller":"sharedmain/main.go:185","msg":"Starting controllers...","commit":"e79eab7"}
...

[aslom]

After talking @lionelvillard one idea to consider for future: allow channel dispatcher to be namespace bound ksvc (and not shared across namespaces) as that should improve scalability to have separately scaled dispatchers and make accounting easier for multi-tenant setups?

[aslom]

I think found the right place to disable metrics server however currently it seems to be required?

Removed

          - name: METRICS_DOMAIN
            value: knative.dev/inmemorychannel-dispatcher

but that leas to panic - going to do quick code change to make metrics optional so ksvc diaptcher can start without requiring port 9090 (and confilicting with serving)

(base) aslom@m eventing (channels-ksvc) $ k -n knative-eventing logs imc-dispatcher-7rd8t-deployment-8555bfd5d-tprs9 -c user-container
2019/11/11 20:52:46 Registering 3 clients
2019/11/11 20:52:46 Registering 2 informer factories
2019/11/11 20:52:46 Registering 1 informers
2019/11/11 20:52:46 Registering 1 controllers
{"level":"info","ts":"2019-11-11T20:52:46.442Z","caller":"logging/config.go:108","msg":"Successfully created the logger.","knative.dev/jsonconfig":"{\n  \"level\": \"info\",\n  \"development\": false,\n  \"outputPaths\": [\"stdout\"],\n  \"errorOutputPaths\": [\"stderr\"],\n  \"encoding\": \"json\",\n  \"encoderConfig\": {\n    \"timeKey\": \"ts\",\n    \"levelKey\": \"level\",\n    \"nameKey\": \"logger\",\n    \"callerKey\": \"caller\",\n    \"messageKey\": \"msg\",\n    \"stacktraceKey\": \"stacktrace\",\n    \"lineEnding\": \"\",\n    \"levelEncoder\": \"\",\n    \"timeEncoder\": \"iso8601\",\n    \"durationEncoder\": \"\",\n    \"callerEncoder\": \"\"\n  }\n}\n"}
{"level":"info","ts":"2019-11-11T20:52:46.442Z","caller":"logging/config.go:109","msg":"Logging level set to info"}
{"level":"info","ts":"2019-11-11T20:52:46.447Z","logger":"inmemorychannel_dispatcher.in-memory-channel-dispatcher","caller":"dispatcher/controller.go:86","msg":"Setting up event handlers","commit":"e79eab7","knative.dev/controller":"in-memory-channel-dispatcher"}
{"level":"info","ts":"2019-11-11T20:52:46.448Z","logger":"inmemorychannel_dispatcher","caller":"profiling/server.go:60","msg":"Profiling enabled: false","commit":"e79eab7"}
panic: The environment variable "METRICS_DOMAIN" is not set

If this is a process running on Kubernetes, then it should be specifying
this via:

  env:
  - name: METRICS_DOMAIN
    value: knative.dev/some-repository

If this is a Go unit test consuming metric.Domain() then it should add the
following import:

import (
	_ "knative.dev/pkg/metrics/testing"
)

goroutine 1 [running]:
knative.dev/eventing/vendor/knative.dev/pkg/metrics.Domain(0x19, 0xc00004204a)
	/Users/aslom/Documents/awsm/go/src/knative.dev/eventing/vendor/knative.dev/pkg/metrics/config.go:266 +0xfa
knative.dev/eventing/vendor/knative.dev/pkg/metrics.UpdateExporterFromConfigMap(0x191b947, 0x1a, 0xc0000bd020, 0x14)
	/Users/aslom/Documents/awsm/go/src/knative.dev/eventing/vendor/knative.dev/pkg/metrics/exporter.go:67 +0x26
knative.dev/eventing/vendor/knative.dev/pkg/injection/sharedmain.MainWithConfig(0x1caf8c0, 0xc0000bc050, 0x191b947, 0x1a, 0xc0001961e0, 0xc000583f80, 0x1, 0x1)
	/Users/aslom/Documents/awsm/go/src/knative.dev/eventing/vendor/knative.dev/pkg/injection/sharedmain/main.go:171 +0xcb1
knative.dev/eventing/vendor/knative.dev/pkg/injection/sharedmain.MainWithContext(0x1caf8c0, 0xc0000bc050, 0x191b947, 0x1a, 0xc000583f80, 0x1, 0x1)
	/Users/aslom/Documents/awsm/go/src/knative.dev/eventing/vendor/knative.dev/pkg/injection/sharedmain/main.go:111 +0x21d
knative.dev/eventing/vendor/knative.dev/pkg/injection/sharedmain.Main(0x191b947, 0x1a, 0xc000583f80, 0x1, 0x1)
	/Users/aslom/Documents/awsm/go/src/knative.dev/eventing/vendor/knative.dev/pkg/injection/sharedmain/main.go:97 +0x9d
main.main()
	/Users/aslom/Documents/awsm/go/src/knative.dev/eventing/cmd/in_memory/channel_dispatcher/main.go:29 +0x67

[aslom]

I could not find a diagram describing Knative data-plane architecture - do you know any good blog post? links?

I have put together a simple diagram showing current architecture:

and what it would look with ksvc and scaling:

You can edit it here:
https://docs.google.com/presentation/d/13V8EE9nHEgyKrrKMMvUdeSymWFeEdRFCg_qv7n-7QRo/edit?usp=sharing

[mikehelmick]

related to #1555