[Experimental] DeliverySpec RetryAfter

[travis-minke-sap]

Description
The Retry-After header is a standard part of the HTTP spec which can be returned with 429 and 503 responses in order to specify a duration, or timestamp, which subsequent retries should wait before attempting to resend. It provides downstream event recipients with a mechanism to provide back-pressure when requests are arriving too frequently. The event retry mechanism in Knative eventing currently does not respect the Retry-After header. The intent of this new experimental-feature is to expose the ability for users to opt-in to respecting the Retry-After header.

Design Overview
Following the pattern established in the experimental-features process, and closely mirroring the implementation in the similar Delivery Timeout experimental-feature, the plan is to enhance the DeliverySpec to include a new optional retryAfter component. Use of this new component will be gated by the delivery-retryafter experimental feature flag in the config-features ConfigMap and enforced via WebHook validation.

Example DeliverySpec with new retryAfter component...

delivery:
  backoffDelay: PT0.5S
  backoffPolicy: exponential
  retry: 3
  retryAfter:
    enabled: true
    maxDuration: "PT30S"

The new retryAfter component will only take effect if the retry value is specified and is at least 1. The optional maxDuration field provides an override to prevent excessive backoff durations as might be desirable in certain use cases.

Exit Criteria
DeliverySpec allows optional configuration of retry behavior for 429 and 503 Retry-After headers.

Experimental Feature Flag Name
delivery-retryafter

Experimental Feature Stages
The following is the proposed plan for moving through the stages of the experimental feature process...

• Alpha: (Target 1.1 Release)
  • Initial Implementation, Unit Tests, e2e Test [Experimental-Feature] DeliverySpec.RetryAfter #5813
  • User Documentation DeliverySpec.RetryAfter Experimental-Feature docs#4361
  • Refactor Flaky Unit Tests Fix flaky Retry-After Header Test #5943, Refactor Retry-After backoff tests #5981
  • Fix Channel Backed MT Broker To Proxy Response Headers Enhance Broker Filter to proxy Response Headers #5946
  • Adjust KafkaChannel CRD to allow experimental-features in delivery spec PR template should have examples of good release notes #1025.
• Beta: (minimum of 1 release after Alpha)
  • Changes / Fixes based on Alpha ?
  • Documentation Enhancements ?
  • Conformance Tests ?
• Stable: (minimum of 2 releases after Beta)
  • Remove delivery-retryafter feature flag.
  • Remove eventing WebHook validation enforcement.
  • Add requirements to support DeliverySpec.RetryAfter to knative/specs repo

Affected WG

• Eventing WG
• Eventing-Kafka WG (previously Event-Delivery)

Additional Context

• Resources
  Experimental Features Process
  Retry-After HTTP 1.1 RFC
  Retry-After Mozilla Docs
  CloudEvent Webhook Spec

• History
  An initial attempt at supporting Retry-After headers was made in March 2021 and is mostly documented in Knative Eventing Discussion #5011. This second attempt has been briefly discussed at the Eventing WG in the past few weeks.

[travis-minke-sap]

/assign

[evankanderson]

Do we need the enabled bool, or could we require a max duration (even if it's something silly like "1 week")?

Other questions:

1. Any guidance to implementers about handling this in-process vs in the storage layer?

2. Should a Retry-After apply to all deliveries to that host, or only that specific delivery?

3. Should there be an administrator-level ConfigMap parameter to specify a max retry-after duration?

[evankanderson]

(I'm 👍 on this feature, by the way)

[travis-minke-sap]

Thanks for the 👀 's on this Evan - much appreciated! I'll do my best to respond, forgive me if I miss the intent - I don't have as clear an understanding of the larger eventing ecosystem as you do (too narrowly focused on the KafkaChannel ; )

Also, I was holding off on creating the PR (I probably got ahead of myself by doing the work before the issue was reviewed) until I had the e2e tests were done... but I've gone ahead and created it so that folks can see the intended changes. I am still happy to refactor as the community deems necessary ; )

Do we need the enabled bool, or could we require a max duration (even if it's something silly like "1 week")?

I would vote to keep the separation, but am certainly open to that approach if the community prefers it. I only added maxDuration as an afterthought safety gauge, and wasn't sure what the reaction or need would be. It feels a bit wrong to combine the two and require a maxDuration in all cases, and more importantly for it to "infer" the enabled state. Just my $0.02 though - we can see if there are any other opinions ; )

1. Any guidance to implementers about handling this in-process vs in the storage layer?

I'm not sure what you mean by "handling in the storage layer"? My intent (and what I've implemented thus far) is to handle this exclusively "in-process" on a per-event basis. Meaning, when a dispatcher sends an event and receives back a 429/Retry-After, it would apply that backoff before resending that event only.

2. Should a Retry-After apply to all deliveries to that host, or only that specific delivery?

As I mentioned in 1. above, I see this as being used to inform the determination of the backoff duration for a single event request only. This is what I've implemented in the message_sender.go SendWithRetries() implementation. There is nothing about this new experimental-feature that would prevent a custom implementation from taking a broader view and throttling other parallel requests.

3. Should there be an administrator-level ConfigMap parameter to specify a max retry-after duration?

I haven't seen any precedent for similar DeliverySpec changes (e.g. Timeout field) and wouldn't know where such a thing would live? I would rather wait and see if there is request for global config and add it in later if necessary.

[matzew]

If I set retry: 3 (or any larger as 0) I do indicate that I am interested in "retrying".

So, for me I'd thnk, that I am also interested in doing a retry-after recognition, for a good reason.

Therefore not really sure if we need to tweak it and add extra nobs, like enabling/disabling it.

However, I can see that if the server indicates 600 seconds, that this might be too long for me...

[matzew]

(I'm 👍 on this feature too, by the way)

[travis-minke-sap]

If I set retry: 3 (or any larger as 0) I do indicate that I am interested in "retrying".

So, for me I'd thnk, that I am also interested in doing a retry-after recognition, for a good reason.

Therefore not really sure if we need to tweak it and add extra nobs, like enabling/disabling it.

However, I can see that if the server indicates 600 seconds, that this might be too long for me...

Good point @matzew - I think the question you're really asking is... "If/when this feature graduates to Stable/GA, would we want to allow users of the retry capability an option to respect/ignore Retry-After headers?" If we are comfortable answering "No, if you use retry then Retry-After headers WILL be respected", then the DeliverySpec configuration can be reduced to a single optional field as follows...

delivery:
  backoffDelay: PT0.5S
  backoffPolicy: exponential
  retry: 3
  retryAfterMax: PT120S

...and during the experimental-features Alpha/Beta stages the feature flag would simply control whether or not to respect the Retry-After headers, instead of gating the DeliverySpec.RetryAfter configuration during validation.

Personally, I'm fine with this approach as it seems to better adhere to the intentions of the HTTP Spec, but I thought this was undesirable from a "don't change the existing behavior" perspective. It's unclear how many users would actually try out the experimental-feature between Alpha/Stable, so it's possible the switchover might catch them out?

[matzew]

On Thu 14. Oct 2021 at 21:22, Travis Minke ***@***.***> wrote: If I set retry: 3 (or any larger as 0) I do indicate that I am interested in "retrying". So, for me I'd thnk, that I am also interested in doing a retry-after recognition, for a good reason. Therefore not really sure if we need to tweak it and add extra nobs, like enabling/disabling it. However, I can see that if the server indicates 600 seconds, that this might be too long for me... Good point @matzew <https://github.com/matzew> - I think the question you're really asking is... *"If/when this feature graduates to Stable/GA, would we want to allow users of the retry capability an option to respect/ignore Retry-After headers?"* If we are comfortable answering *"No, if you use retry then Retry-After headers WILL be respected"*, then the DeliverySpec configuration can be reduced to a single optional field as follows... delivery: backoffDelay: PT0.5S backoffPolicy: exponential retry: 3 retryAfterMax: PT120S

more compact and reasonable to express it like this ...and during the experimental-features Alpha/Beta stages the feature flag

would simply control whether or not to respect the *Retry-After* headers, instead of gating the DeliverySpec.RetryAfter configuration during validation. Personally, I'm fine with this approach as it seems to better adhere to the intentions of the HTTP Spec,

ACK, that’s right. Especially if I set retry to “n” (larger 0) but I thought this was undesirable from a *"don't change the existing

behavior"* perspective. It's unclear how many users would actually try out the experimental-feature between Alpha/Stable, so it's possible the switchover might catch them out?

I’d use it: if I can honor the retry-after header I’d like to do so. IMO an oversight when designing retry —

You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5811 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABGPTTATM6NJN2BBSSAM7DUG4UURANCNFSM5F6AMC6Q> .

-- Sent from Gmail Mobile

[travis-minke-sap]

I’d use it: if I can honor the retry-after header I’d like to do so.
IMO an oversight when designing retry

I like it! I can re-tool to that implementation if folks agree (or no-one disagrees after a week and a summary at next weeks WG) ?

[lionelvillard]

+1 for retryAfterMax. The default behavior should be to always respect retry-after.

Setting retryAfterMax: PT0S provides an effective way to disable this feature. We could provide a post-install job to set this field, as a way to make sure that existing applications won't break when upgrading.

[pierDipi]

+1 for retryAfterMax. The default behavior should be to always respect retry-after.

+1.