handle overflow in TaskInstance next_retry_datetime fixes 47971

[perry2of5]

This handles overflow when calculating the next execution time for a task instance by falling back to the configured maximum delay. The solution uses the same strategy that tenacity uses:
https://github.com/jd/tenacity/blob/main/tenacity/wait.py#L167

An alternate solution would be the determine the maximum tries that wouldn't exceed the maximum delay and then not calculate the timeout for values larger than that.

Something like

max_delay = self.task.max_retry_delay if self.task.max_retry_delay is not null else MAX_RETRY_DELAY
tries_before_max_delay = math.floor(math.log2(max_delay))
if self.try_number <= tries_before_max_delay:
     # existing logic
else:
    delay = max_delay

closes: #47971

[perry2of5]

If you turn off whitespace changes, only 6 lines are added in the new version and none are removed. :)

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

[uranusjr]

Is it possible to localise the try-except block a bit more? Currently the block is a bit too large IMO.

[perry2of5]

The overflow error is likely two places:

on line 1120:
min_backoff = math.ceil(delay.total_seconds() * (2 ** (self.try_number - 1)))

And on line 1145: https://docs.python.org/3/library/datetime.html#timedelta-objects
delay = timedelta(seconds=delay_backoff_in_seconds)

Keeping the try block as it currently is avoids having two try blocks, but possibly two try blocks would be more readable. I'll fiddle with it and try to improve. Thanks for the feedback.

[perry2of5]

Looking a little closer, we cap MAX_RETRY_DELAY at 24 * 60 * 60 so probably it can't throw at line 1145.... I'll need to be sure that isn't configurable at run time, but as long as it is always less than 999999999 days then I can shorten up the section in the try block. I'll try to get to it tonight.

[perry2of5]

Is it possible to localise the try-except block a bit more? Currently the block is a bit too large IMO.

Done. Thanks for the feedback, it is much cleaner now.

[perry2of5]

FWIW, I think we should drop the log when overflow is caught since we don't log other reasons for capping the delay.

[amoghrajesh]

@uranusjr could you take a look at this PR again when you have some time?

[github-actions]

Backport successfully created: v3-0-test

Status	Branch	Result
✅	v3-0-test