[SPARK-3597][Mesos] Implement killTask.

[brndnmtthws]

The MesosSchedulerBackend did not previously implement killTask,
resulting in an exception.

[rxin]

Sorry for asking - but have you tested this on a real cluster?

[rxin]

Oh and thanks for doing this!

[brndnmtthws]

I did indeed test it, and it worked as expected.

[SparkQA]

Can one of the admins verify this patch?

[andrewor14]

ok to test

[andrewor14]

There is a related PR #1940

[andrewor14]

test this please

[SparkQA]

QA tests have started for PR 2453 at commit 40c4478.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2453 at commit 40c4478.

• This patch fails unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class IDF(val minDocFreq: Int)
  • class DocumentFrequencyAggregator(val minDocFreq: Int) extends Serializable
  • class PStatsParam(AccumulatorParam):

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20872/

[brndnmtthws]

Build error appears to be unrelated to my patch.

[JoshRosen]

Build failure is my fault, due to merging #2351. I've reverted it and re-triggered your unit tests.

[brndnmtthws]

Okay, I'll rebase.

[SparkQA]

QA tests have started for PR 2453 at commit 40c4478.

• This patch merges cleanly.

[SparkQA]

QA tests have started for PR 2453 at commit 23ddcdc.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2453 at commit 40c4478.

• This patch passes unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class IDF(val minDocFreq: Int)
  • class DocumentFrequencyAggregator(val minDocFreq: Int) extends Serializable

[SparkQA]

QA tests have finished for PR 2453 at commit 23ddcdc.

• This patch passes unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20882/

[brndnmtthws]

This code has been tested & verified on a cluster of this size:

[andrewor14]

LGTM, but I'm no expert on the mesos integration. @tnachen any comments?

[JoshRosen]

This looks good to me. I tested this patch against Mesos 0.20.1 running in Docker with a modified version of a test from Spark's JobCancellationSuite .

@tnachen commented on this over at #1940 (comment):

On Mesos side if you call killTask on a non-existing Task all you get is a LOG(WARNING).

[andrewor14]

Alright, thanks for testing @JoshRosen and @brndnmtthws. I'm merging this into master and 1.1.