Decide backwards compatibility requirements for Channels CRD changes

[nachocano]

Problem
As we are moving away from the channel provisioner model towards the operator one, there is a need to decide what is the backwards compatibility we want to maintain for the provisioner-based channels, if any.

Exit Criteria
Decision on how to address backwards compatibility for moving the channels to CRDs.

Time Estimate (optional):
1-2 days.

Additional context (optional)
This will have an impact in #1214, #1216, #1217, #1215

[nachocano]

/assign

[nachocano]

AFAIK the InMemory channel changes introduced a new CRD, but didn't touch the existing provisioner-based model . Therefore, the two can co-exist, there will be two controllers. I'm not entirely sure if the idea is to mark the provisioner model as deprecated in 0.7, and remove it in 0.8? Or just get rid of it in 0.7.

A similar decision needs to be made for the other Channels. I'm starting to work on GcpPubSub and I'm following the same strategy as the InMemory one, i.e., leaving the provisioner as is, and creating a new controller for the CRD-based one.

Talking with @akashrv we agreed this needs further discussion, thus I opened this issue.

[nachocano]

/cc @vaikas-google @Harwayne @grantr @evankanderson
Need your input on this. The sooner we can agree on this, the better, so that we can use the same approach in all of the other channels.

[Harwayne]

If we intend to run both CRDs and Provisioners in parallel for a long time, then I would want to use a single controller. If we plan to remove the Provisioners quickly, then I am fine having them be completely distinct controllers.

A possible hybrid is to use new CRD controllers, then alter the provisioner controllers to proxy CRD<->Channel with Provisioner. That way the behavior is identical, even with two distinct controllers.

---

A more general question I have is about migration. As users migrate from provisioners to CRDs, are events allowed to drop? Or do we need to support a migration without any events dropping? Or in a slightly weaker statement, are events allowed to come out of order?

Dropped example:

1. Delete existing Provisioner based channel.
2. Make a new CRD channel.
3. Hook up all the Subscriptions to the new CRD channel.
4. Move all references from the provisioner based channel to the CRD channel.

Out of order example:

1. Make a new CRD channel.
2. Duplicate all the Subscriptions from the old provisioner based Channel onto the CRD channel.
3. Move all references from the provisioner based channel to the CRD channel.
4. Wait until the provisioner channel totally drains of all events.
5. Delete the provisioner based channel.

We could try to be very careful and reuse the existing underlying abstractions in the new CRD channels. E.g. make sure the new PubSub CRD channel uses the same underlying GCP PubSub Topic as the old Provisioner channel. Although this is likely very error prone and requires the new controller to understand and migrate the old provisioner channels.

[nachocano]

Another unrelated but relevant question is whether we should be using controller-runtime for Channel controllers. All but the new InMemory CRD one use controller-runtime. IIRC all but the InMemory one were going to be moved to eventing-contrib (once eventing-sources is renamed). In the current eventing-sources we are still using controller-runtime.

[nachocano]

If we plan to remove the Provisioners quickly, then I am fine having them be completely distinct controllers.

I think that is the plan, but I'm not sure. I'll vote for this option....
Also, if we have two completely distinct controllers, we can start writing them with the serving infra we introduced in 0.6 to unify the controllers (not controller-runtime)... But I guess that will depend on whether we are planning on moving eventing-sources to that infra, as it seems that sources and channels will end up living together in eventing-contrib.

[akashrv]

If we intend to run both CRDs and Provisioners in parallel for a long time, then I would want to use a single controller. If we plan to remove the Provisioners quickly, then I am fine having them be completely distinct controllers.

A possible hybrid is to use new CRD controllers, then alter the provisioner controllers to proxy CRD<->Channel with Provisioner. That way the behavior is identical, even with two distinct controllers.

A more general question I have is about migration. As users migrate from provisioners to CRDs, are events allowed to drop? Or do we need to support a migration without any events dropping? Or in a slightly weaker statement, are events allowed to come out of order?

Dropped example:

1. Delete existing Provisioner based channel.
2. Make a new CRD channel.
3. Hook up all the Subscriptions to the new CRD channel.
4. Move all references from the provisioner based channel to the CRD channel.

Out of order example:

1. Make a new CRD channel.
2. Duplicate all the Subscriptions from the old provisioner based Channel onto the CRD channel.
3. Move all references from the provisioner based channel to the CRD channel.
4. Wait until the provisioner channel totally drains of all events.
5. Delete the provisioner based channel.

We could try to be very careful and reuse the existing underlying abstractions in the new CRD channels. E.g. make sure the new PubSub CRD channel uses the same underlying GCP PubSub Topic as the old Provisioner channel. Although this is likely very error prone and requires the new controller to understand and migrate the old provisioner channels.

My two cents:
If we are moving to a newer model, then we should deprecate the older one asap esp. because we have the liberty to do so in alpha. Keeping both running side-by-side just increases the tax to support both - Deprecate provisioners in 0.7, remove provisioners n 0.8.
Moreover, are there production customers that we will break and is it worth all the extra work/effort?
If it is about building the expertise to roll-out such breaking changes in future, then a design doc on how we would do it is sufficient.

[nachocano]

Deprecate provisioners in 0.7, remove provisioners n 0.8.

+1

are there production customers that we will break and is it worth all the extra work/effort?

I don't think so. IMO is not worth the extra effort, but I might be wrong.

[vaikas]

My preference is to create a new separate controller and we can keep the existing ones as they are. This is what I did with the in-memory one also.

[nachocano]

Upon some offline discussions, we agreed that we will have the two controllers coexist, and mark the provisioner-based one deprecated in 0.7. They will be removed in 0.8 and we will just keep the operator-based ones.

Also, the new channel controllers should use knative/pkg controller infra instead of controller-runtime.