KAFKA-7790: Fix Bugs in Trogdor Task Expiration#6103
Merged
cmccabe merged 5 commits intoapache:trunkfrom Jan 11, 2019
stanislavkozlovski:trogdor-timeout
Merged
KAFKA-7790: Fix Bugs in Trogdor Task Expiration#6103cmccabe merged 5 commits intoapache:trunkfrom stanislavkozlovski:trogdor-timeout
cmccabe merged 5 commits intoapache:trunkfrom
stanislavkozlovski:trogdor-timeout
Conversation
This commit changes a Trogdor agent/coordinator's behavior to not run tasks that have expired. We define an expired task as one whose sum of `startedMs` and `durationMs` is less than the current time in milliseconds.
| try { | ||
| client.createWorker(new CreateWorkerRequest(workerId, taskId, spec)); | ||
| if (startedMs == -1) | ||
| startedMs = time.milliseconds(); |
Contributor
Author
There was a problem hiding this comment.
Here the Coordinator keeps track of when it has first ran the task - useful in cases where spec.startMs==0
| if (!worker.hasExpired()) { | ||
| worker.tryCreate(); | ||
| } else { | ||
| log.info("{}: Will not create worker state {} as it has expired. ", node.name(), worker.state); |
Contributor
Author
There was a problem hiding this comment.
This is in the case where a Coordinator re-schedules a task when an Agent is detected via the heartbeats
| } catch (Throwable t) { | ||
| failure = "Failed to create TaskController: " + t.getMessage(); | ||
| } | ||
| if (spec.hasExpired(time, -1)) |
Contributor
Author
There was a problem hiding this comment.
This is where a Coordinator is given a brand new task that is expired
…nt NPathComplexity
Contributor
Author
|
Retest this please |
cmccabe
reviewed
Jan 9, 2019
| public boolean hasExpired(Time time, long startedMs) { | ||
| long startMs = this.startMs > 0 ? this.startMs : startedMs; | ||
| if (startMs <= 0) // task doesn't have a start time yet | ||
| return false; |
Contributor
There was a problem hiding this comment.
We shouldn't be special-casing 0 here. In general, I think the only thing we need here is an accessor function like endMs which returns startMs + durationMs.
cmccabe
reviewed
Jan 9, 2019
| this.reference = shutdownManager.takeReference(); | ||
| } | ||
|
|
||
| boolean hasExpired() { |
Contributor
There was a problem hiding this comment.
I don't know if it makes sense to add this function just for a single caller.
We could just check if (spec.endMs() >= time.milliseconds()) below...
Also reverts back changes to NodeManager
…t NPathComplexity
cmccabe
changed the title
cmccabe
changed the title
Contributor
|
retest this please |
Contributor
Author
|
JDK11 failure seems unrelated - |
Contributor
|
LGTM |
pengxiaolong
pushed a commit
to pengxiaolong/kafka
that referenced
this pull request
Jun 14, 2019
The Trogdor Coordinator now overwrites a task's startMs to the time it received it if startMs is in the past. The Trogdor Agent now correctly expires a task after the expiry time (startMs + durationMs) passes. Previously, it would ignore startMs and expire after durationMs milliseconds of local start of the task. Reviewed-by: Colin P. McCabe <cmccabe@apache.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
https://issues.apache.org/jira/browse/KAFKA-7790
Changes:
startMsto the time it received it ifstartMsis in the past.startMs + durationMs) passes. Previously, it would ignorestartMsand expire afterdurationMsmilliseconds of local start of the task.