retrigger failed test tasks for Linux cloud pools 5 times and identify permanent failures

[Archaeopteryx]

Subtask of #1189

If a task failed and has the jobKind of test:

• Check if the task has been classified. If yes, do nothing and quit here.
• Check if the task is tier 1 or 2. If not, do nothing and quit here.
• Identify if the failure are test failures or other issues, e.g. infrastructure
  • If there is no task artifact which contains errorsummary in its name or is incomplete, then the task either hit an infrastructure issue or the test suite does not support failure summaries by errorsummary.
• If test failure
  • Check if the task got executed on Linux cloud worker pool. The Task Queue ID listed for it (example should start with gecko-t/t-linux-.
  • Schedule up to 5 task retriggers to have at least 6 tasks (original one + 5 retriggers): check if there are already retriggers scheduled for the push
• retriggers complete: Decide if the failure rate is a regression
  • retriggers: At least 2 retriggers observed the same test failures as the initial task: regression

Classification would be automatically by LPID.

[Archaeopteryx]

Addition: Don't retrigger tasks which have already been classified.

[vrigal]

@Archaeopteryx I make a first draft implementation in #1196, so we can discuss the details there.

[vrigal]

@Archaeopteryx do you have an idea how to compare if a failed retrigger is the same issue than the original task ?
We don't have access to the logs from Mozci, and I suppose it's not a reliable way to compare the failure.

Should we consider any retrigger failure as a similar failure ?

[Archaeopteryx]

Recommendation:

1. For the initially failed task, fetch the bug_suggestions data from Treeherder (example).
2. Get all array elements (= failure lines) for which search starts with TEST-UNEXPECTED-.
3. Count how often these failures have been observed in the retriggers. This likely needs some delay because Treeherder needs to parse the logs first. The alternative is to use the .errorsummary..json artifact and only compare the failed manifests (test folders) but it has the risk to be too noisy, especially for platforms where tests execute slowly (ASan, TSan).

[vrigal]

1. For the initially failed task, fetch the `bug_suggestions` data from Treeherder ([example](https://treeherder.mozilla.org/api/project/mozilla-central/jobs/530611527/bug_suggestions/)).

I suppose we can fetch the Treeherder job ID using https://treeherder.mozilla.org/api/jobs/?task_id=<task_id>, then rely on bug_suggestion for comparison (it seems a good design to rely on Treeherder IMO). As you mentioned there could be race conditions here, so maybe we can use the endpoint I mentioned to ensure all the retriggers have a job ID in Treeherder before continuing ?

I would like also some clarification about how the push classification command is called. My understanding is that it will be called once initially to start the retriggers, then it will be called a few times later on, to perform the backfills (once all the retriggers are completed and present in Treeherder). If I remember correctly the command is called from a cron-like task, for all the recent pushes within a given date range, right ?

[Archaeopteryx]

Yes.

1. Check if Treeherder has ingested the task, retrieve Treeherder's job ID for it.
2. Check if the job logs have been parsed (javascript consumer example): https://treeherder.mozilla.org/api/project/autoland/job-log-url/?job_id=JOB_ID
3. If logs have been parsed, request the bug suggestions.

[marco-c]

I would like also some clarification about how the push classification command is called. My understanding is that it will be called once initially to start the retriggers, then it will be called a few times later on, to perform the backfills (once all the retriggers are completed and present in Treeherder). If I remember correctly the command is called from a cron-like task, for all the recent pushes within a given date range, right ?

Correct. The code that is running on a cron is https://github.com/mozilla/mozci/blob/master/mozci/console/commands/decision.py.