Skip to content
This repository was archived by the owner on Apr 20, 2026. It is now read-only.

fix never hitting the "TooLate (MissedStart)" condition#16

Merged
yangkev merged 1 commit intorelease-1.14.10-lyftfrom
yangkev-test-toolate-patch
Apr 1, 2020
Merged

fix never hitting the "TooLate (MissedStart)" condition#16
yangkev merged 1 commit intorelease-1.14.10-lyftfrom
yangkev-test-toolate-patch

Conversation

@yangkev
Copy link
Copy Markdown

@yangkev yangkev commented Mar 30, 2020

This issue was first raised in kubernetes#73169.

In the past, we've seen that when the cronjob controller's syncAll loop
time exceeds startingDeadlineSeconds, deadlines get missed.
However, we currently do not get any notifications or telemetry on when
deadlines are missed.

Allowing the MissedStart event emission code path is the first step
to getting telemetry on missed deadlines.

There have been attempts to fix kubernetes#73169 by removing the "TooLate
(MissedStart)" check wholesale:

  1. Fix CronJob missed start time handling kubernetes/kubernetes#81557 (open)
  2. Fix Code related to CronJob is unreachable kubernetes/kubernetes#74058 (closed)

I don't believe this check should be removed since the user should be
notified when their deadline is missed.

Changes:

  • test when startingDeadline elapses, should count as an unmet start time

  • wrap table driven tests in t.Run to allow running individual tests

  • make tests expect MissedStart event when startingDeadline is missed

  • fix TooLate (MissedStart) condition never being hit

@yangkev
Copy link
Copy Markdown
Author

yangkev commented Mar 30, 2020

For review, choose the "split" option, and "hide whitespace changes".

Comment thread pkg/controller/cronjob/cronjob_controller_test.go Outdated
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This "count the events" part of the test is starting to get a little goofy, but I guess it's not worth ripping the bandage off now.

@vllry
Copy link
Copy Markdown

vllry commented Mar 30, 2020

/lgtm

@yangkev yangkev force-pushed the yangkev-test-toolate-patch branch from fbf9850 to 542f1dc Compare March 30, 2020 21:08
@yangkev yangkev changed the base branch from release-1.14.10-lyft to yangkev-make-tests-table-driven March 30, 2020 21:09
@yangkev yangkev force-pushed the yangkev-test-toolate-patch branch from 542f1dc to 81ea72b Compare March 30, 2020 23:58
@yangkev yangkev requested a review from vllry March 31, 2020 17:00
@yangkev yangkev force-pushed the yangkev-test-toolate-patch branch from 81ea72b to ad3088c Compare March 31, 2020 23:53
Comment thread pkg/controller/cronjob/cronjob_controller_test.go Outdated
@vllry
Copy link
Copy Markdown

vllry commented Apr 1, 2020

Still looks good.

@yangkev yangkev changed the base branch from yangkev-make-tests-table-driven to release-1.14.10-lyft April 1, 2020 00:18
This issue was first raised in kubernetes#73169.

In the past, we've seen that when the cronjob controller's syncAll loop
time exceeds startingDeadlineSeconds, deadlines get missed.
However, we currently do not get any notifications or telemetry on when
deadlines are missed.

Allowing the `MissedStart` event emission code path is the first step
to getting telemetry on missed deadlines.

There have been attempts to fix this by removing the "TooLate
(MissedStart)" check wholesale:

1. kubernetes#81557 (open)
2. kubernetes#74058 (closed)

I don't believe this check should be removed since the user should be
notified when their deadline is missed.

Changes:
- test when startingDeadline elapses, should count as an unmet start time

- make tests expect MissedStart event when startingDeadline is missed

- fix TooLate (MissedStart) condition never being hit
@yangkev yangkev force-pushed the yangkev-test-toolate-patch branch from ad3088c to 4928a20 Compare April 1, 2020 00:25
@yangkev yangkev merged commit 25af335 into release-1.14.10-lyft Apr 1, 2020
theatrus pushed a commit that referenced this pull request Apr 7, 2020
This issue was first raised in kubernetes#73169.

In the past, we've seen that when the cronjob controller's syncAll loop
time exceeds startingDeadlineSeconds, deadlines get missed.
However, we currently do not get any notifications or telemetry on when
deadlines are missed.

Allowing the `MissedStart` event emission code path is the first step
to getting telemetry on missed deadlines.

There have been attempts to fix this by removing the "TooLate
(MissedStart)" check wholesale:

1. kubernetes#81557 (open)
2. kubernetes#74058 (closed)

I don't believe this check should be removed since the user should be
notified when their deadline is missed.

Changes:
- test when startingDeadline elapses, should count as an unmet start time

- make tests expect MissedStart event when startingDeadline is missed

- fix TooLate (MissedStart) condition never being hit
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants