Skip to content

Bug 1519374 - Stop using gecko-focus in favor or mobile-X workers #579

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
JohanLorenzo merged 3 commits into from
Feb 14, 2019
Merged

Bug 1519374 - Stop using gecko-focus in favor or mobile-X workers #579

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
8fc7469
Refactor .taskcluster.yml to define common values once
JohanLorenzo Feb 7, 2019
486460d
Switch decision and build worker types
JohanLorenzo Feb 11, 2019
a765a54
Merge branch 'master' into bug-1519374
JohanLorenzo Feb 14, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
342 changes: 189 additions & 153 deletions .taskcluster.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,158 +5,194 @@ version: 1
policy:
pullRequests: public
tasks:
- $if: 'tasks_for == "github-pull-request" && event["action"] in ["opened", "reopened", "synchronize"]'
then:
$let:
decision_task_id: {$eval: as_slugid("decision_task")}
expires_in: {$fromNow: '1 year'}
user: ${event.sender.login}

# We define the following variable at the very top, because they are used in the
# default definition
head_branch:
$if: 'tasks_for == "github-pull-request"'
then: ${event.pull_request.head.ref}
else: ${event.release.target_commitish}

head_rev:
$if: 'tasks_for == "github-pull-request"'
then: ${event.pull_request.head.sha}
else: ${event.release.tag_name}

repository:
$if: 'tasks_for == "github-pull-request"'
then: ${event.pull_request.head.repo.html_url}
else: ${event.repository.html_url}

scheduler_id:
$if: 'tasks_for == "cron"'
then: focus-nightly-sched # TODO: Rename to mobile-nightly-sched
else: taskcluster-github

is_repo_trusted:
# Pull requests on main repository can't be trusted because anybody can open a PR on it, without a review
$if: 'tasks_for in ["github-push", "github-release", "cron"] && event.repository.html_url == "https://github.com/mozilla-mobile/reference-browser"'
then: true
else: false
in:
$let:
decision_worker_type:
$if: 'is_repo_trusted'
Copy link
Copy Markdown
Contributor

@mitchhentges mitchhentges Feb 11, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Having to split these onto two levels because of how Taskcluster handles variables is 😢
I made a bug for this, but it's stalled

Copy link
Copy Markdown
Contributor Author

@JohanLorenzo JohanLorenzo Feb 12, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah :/ I don't think this is a big deal for us, yet. So far, we just need 2 layers. It would become a bigger deal if we had to deal with 4, I think.

Copy link
Copy Markdown
Contributor

@mitchhentges mitchhentges Feb 12, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's unnecessary noise that you need to filter out when scanning through the file, and it throws you off: when I see a block is indented, I implicitly assume that it's a bit "different" from less-indented blocks. Though we haven't reached a "staircase" (:stuck_out_tongue:) amount of steps for $let just yet, it's still a little jarring to have to filter out that "yes this is indented, but it's just more var declarations".

There's not much we can do here, so 👍 :)

then: mobile-3-decision
else: mobile-1-decision

build_worker_type:
$if: 'is_repo_trusted'
then: mobile-3-b-ref-browser
else: mobile-1-b-ref-browser

tasks_priority:
$if: 'is_repo_trusted'
then:
$if: 'tasks_for == "github-release"'
then: highest
else:
$if: 'tasks_for == "cron"'
then: high
else: medium
else: lowest
in:
$let:
decision_task_id: {$eval: as_slugid("decision_task")}
expires_in: {$fromNow: '1 year'}
scheduler_id: taskcluster-github
user: ${event.sender.login}
head_branch: ${event.pull_request.head.ref}
head_rev: ${event.pull_request.head.sha}
repository: ${event.pull_request.head.repo.clone_url}
pull_request_title: ${event.pull_request.title}
pull_request_number: ${event.pull_request.number}
pull_request_url: ${event.pull_request.html_url}
in:
taskId: ${decision_task_id}
taskGroupId: ${decision_task_id}
created: {$fromNow: ''}
expires: ${expires_in}
deadline: {$fromNow: '2 hours'}
provisionerId: aws-provisioner-v1
workerType: github-worker
scopes:
- queue:create-task:aws-provisioner-v1/github-worker
- queue:scheduler-id:${scheduler_id}
payload:
maxRunTime: 7200
image: mozillamobile/android-components:1.15
env:
TASK_ID: ${decision_task_id}
SCHEDULER_ID: ${scheduler_id}
MOBILE_HEAD_REPOSITORY: ${repository}
MOBILE_HEAD_BRANCH: ${head_branch}
MOBILE_HEAD_REV: ${head_rev}
GITHUB_PULL_TITLE: ${pull_request_title}
command:
- /bin/bash
- --login
- -cx
- >-
cd ..
&& git clone ${event.pull_request.head.repo.clone_url}
&& cd reference-browser
&& git config advice.detachedHead false
&& git checkout ${event.pull_request.head.sha}
&& python automation/taskcluster/decision_task.py
features:
taskclusterProxy: true
artifacts:
public/reports:
type: directory
path: /build/reference-browser/app/build/reports
expires: {$fromNow: '1 week'}
metadata:
name: Decision task
description: Building reference browser - triggered by a pull request.
owner: ${event.pull_request.user.login}@users.noreply.github.com
source: ${event.repository.url}
- $if: 'tasks_for == "cron"'
then:
$let:
decision_task_id: {$eval: as_slugid('decision_task')}
expires_in: {$fromNow: '1 year'}
repository: ${event.repository.html_url}
scheduler_id: focus-nightly-sched
is_mozilla_mobile_repo:
$eval: event.repository.html_url == 'https://github.com/mozilla-mobile/reference-browser'
command_staging_flag:
$if: event.repository.html_url == 'https://github.com/mozilla-mobile/reference-browser'
then: ''
else: '--staging'
in:
taskId: ${decision_task_id}
taskGroupId: ${decision_task_id} # Must be explicit because of Chain of Trust
created: {$fromNow: ''}
deadline: {$fromNow: '2 hours'}
expires: ${expires_in}
schedulerId: ${scheduler_id} # Must be explicit because of Chain of Trust
provisionerId: aws-provisioner-v1
# TODO Replace by mobile-3-decision (https://bugzilla.mozilla.org/show_bug.cgi?id=1512631)
workerType: gecko-focus # This workerType has ChainOfTrust enabled
requires: all-completed # Must be explicit because of Chain of Trust
priority: medium
retries: 5
scopes:
$flatten:
default_task_definition:
taskId: ${decision_task_id}
taskGroupId: ${decision_task_id} # Must be explicit because of Chain of Trust
schedulerId: ${scheduler_id}
created: {$fromNow: ''}
deadline: {$fromNow: '2 hours'}
expires: ${expires_in}
provisionerId: aws-provisioner-v1
workerType: ${decision_worker_type}
priority: ${tasks_priority}
requires: all-completed # Must be explicit because of Chain of Trust
retries: 5
scopes:
- queue:create-task:${tasks_priority}:aws-provisioner-v1/${build_worker_type}
- queue:route:statuses
- queue:scheduler-id:${scheduler_id}
- queue:create-task:highest:aws-provisioner-v1/gecko-focus
- project:mobile:reference-browser:releng:signing:format:autograph_apk_reference_browser
- secrets:get:project/mobile/reference-browser/sentry
- $if: is_mozilla_mobile_repo
then:
- queue:create-task:highest:scriptworker-prov-v1/mobile-signing-v1
- queue:create-task:highest:scriptworker-prov-v1/mobile-pushapk-v1
- project:mobile:reference-browser:releng:signing:cert:release-signing
- project:mobile:reference-browser:releng:googleplay:product:reference-browser
- queue:route:index.project.mobile.reference-browser.signed-nightly.*
else:
- queue:create-task:highest:scriptworker-prov-v1/mobile-signing-dep-v1
- queue:create-task:highest:scriptworker-prov-v1/mobile-pushapk-dep-v1
- project:mobile:reference-browser:releng:signing:cert:dep-signing
- project:mobile:reference-browser:releng:googleplay:product:reference-browser:dep
- queue:route:index.project.mobile.reference-browser.staging-signed-nightly.*
routes:
- statuses # Automatically added by taskcluster-github. It must be explicit because of Chain of Trust
payload:
maxRunTime: 600 # Decision should remain fast enough to schedule a handful of tasks
image: mozillamobile/android-components:1.15
features:
taskclusterProxy: true
chainOfTrust: true
env:
TASK_ID: ${decision_task_id}
SCHEDULER_ID: ${scheduler_id}
MOBILE_HEAD_REPOSITORY: ${repository}
MOBILE_HEAD_BRANCH: ${event.release.target_commitish}
MOBILE_HEAD_REV: ${event.release.tag_name}
MOBILE_TRIGGERED_BY: ${event.sender.login}
command:
- /bin/bash
- --login
- -cx
- >-
cd ..
&& git clone ${repository}
&& cd reference-browser
&& git checkout ${event.release.tag_name}
&& python automation/taskcluster/decision_task_nightly.py \
--commit \
--output app/build/outputs/apk \
--apk geckoNightlyX86/release/app-geckoNightly-x86-release-unsigned.apk \
--apk geckoNightlyArm/release/app-geckoNightly-arm-release-unsigned.apk \
--apk geckoNightlyAarch64/release/app-geckoNightly-aarch64-release-unsigned.apk \
--date ${now} \
${command_staging_flag}
artifacts:
public/task-graph.json:
type: file
path: /build/reference-browser/task-graph.json
expires: ${expires_in}
public/actions.json:
type: file
path: /build/reference-browser/actions.json
expires: ${expires_in}
public/parameters.yml:
type: file
path: /build/reference-browser/parameters.yml
expires: ${expires_in}
extra:
cron: {$json: {$eval: 'cron'}}
tasks_for: ${tasks_for}
metadata:
name: Decision task (nightly)
description: Scheduling tasks for nightly release of reference browser
owner: android-components-team@mozilla.com
source: ${repository}/raw/${event.release.tag_name}/.taskcluster.yml
routes:
- statuses # Automatically added by taskcluster-github. It must be explicit because of Chain of Trust
payload:
maxRunTime: 600 # Decision should remain fast enough to schedule a handful of tasks
image: mozillamobile/android-components:1.15
command:
- /bin/bash
- --login
- -cx
# The rest of the command must be defined below
env:
TASK_ID: ${decision_task_id}
TASKS_PRIORITY: ${tasks_priority}
SCHEDULER_ID: ${scheduler_id}
BUILD_WORKER_TYPE: ${build_worker_type}
MOBILE_HEAD_REPOSITORY: ${repository}
MOBILE_HEAD_BRANCH: ${head_branch}
MOBILE_HEAD_REV: ${head_rev}
MOBILE_TRIGGERED_BY: ${user}
features:
taskclusterProxy: true
extra:
tasks_for: ${tasks_for}
metadata:
owner: ${user}@users.noreply.github.com
source: ${repository}/raw/${head_rev}/.taskcluster.yml
in:
- $if: 'tasks_for == "github-pull-request" && event["action"] in ["opened", "reopened", "synchronize"]'
then:
$let:
pull_request_title: ${event.pull_request.title}
pull_request_number: ${event.pull_request.number}
pull_request_url: ${event.pull_request.html_url}
in:
$mergeDeep:
- {$eval: 'default_task_definition'}
- payload:
env:
GITHUB_PULL_TITLE: ${pull_request_title}
command:
- >-
cd ..
&& git clone ${repository}
&& cd reference-browser
&& git config advice.detachedHead false
&& git checkout ${head_rev}
&& python automation/taskcluster/decision_task.py
artifacts:
public/reports:
type: directory
path: /build/reference-browser/app/build/reports
expires: {$fromNow: '1 week'}
metadata:
name: Decision task
description: Building reference browser - triggered by a pull request.
- $if: 'tasks_for == "cron"'
then:
$let:
command_staging_flag:
$if: 'is_repo_trusted'
then: ''
else: '--staging'
in:
$mergeDeep:
- {$eval: 'default_task_definition'}
- scopes:
$flatten:
- project:mobile:reference-browser:releng:signing:format:autograph_apk_reference_browser
- secrets:get:project/mobile/reference-browser/sentry
- $if: 'is_repo_trusted'
then:
- queue:create-task:highest:scriptworker-prov-v1/mobile-signing-v1
- queue:create-task:highest:scriptworker-prov-v1/mobile-pushapk-v1
- project:mobile:reference-browser:releng:signing:cert:release-signing
- project:mobile:reference-browser:releng:googleplay:product:reference-browser
- queue:route:index.project.mobile.reference-browser.signed-nightly.*
else:
- queue:create-task:highest:scriptworker-prov-v1/mobile-signing-dep-v1
- queue:create-task:highest:scriptworker-prov-v1/mobile-pushapk-dep-v1
- project:mobile:reference-browser:releng:signing:cert:dep-signing
- project:mobile:reference-browser:releng:googleplay:product:reference-browser:dep
- queue:route:index.project.mobile.reference-browser.staging-signed-nightly.*
payload:
features:
chainOfTrust: true
command:
- >-
cd ..
&& git clone ${repository}
&& cd reference-browser
&& git checkout ${event.release.tag_name}
&& python automation/taskcluster/decision_task_nightly.py \
--commit \
--output app/build/outputs/apk \
--apk geckoNightlyX86/release/app-geckoNightly-x86-release-unsigned.apk \
--apk geckoNightlyArm/release/app-geckoNightly-arm-release-unsigned.apk \
--apk geckoNightlyAarch64/release/app-geckoNightly-aarch64-release-unsigned.apk \
--date ${now} \
${command_staging_flag}
artifacts:
public/task-graph.json:
type: file
path: /build/reference-browser/task-graph.json
expires: ${expires_in}
public/actions.json:
type: file
path: /build/reference-browser/actions.json
expires: ${expires_in}
public/parameters.yml:
type: file
path: /build/reference-browser/parameters.yml
expires: ${expires_in}
extra:
cron: {$json: {$eval: 'cron'}}
tasks_for: ${tasks_for}
metadata:
name: Decision task (nightly)
description: Scheduling tasks for nightly release of reference browser
owner: android-components-team@mozilla.com
4 changes: 3 additions & 1 deletion automation/taskcluster/decision_task.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@
BRANCH = os.environ.get('MOBILE_HEAD_BRANCH')
COMMIT = os.environ.get('MOBILE_HEAD_REV')
PR_TITLE = os.environ.get('GITHUB_PULL_TITLE', '')
BUILD_WORKER_TYPE = os.environ.get('BUILD_WORKER_TYPE', '')


# If we see this text inside a pull request title then we will not execute any tasks for this PR.
SKIP_TASKS_TRIGGER = '[ci skip]'
Expand All @@ -33,7 +35,7 @@ def create_raw_task(name, description, full_command, scopes = []):
deadline = taskcluster.fromNow('1 day')

return {
"workerType": 'github-worker',
"workerType": BUILD_WORKER_TYPE,
"taskGroupId": TASK_ID,
"expires": taskcluster.stringDate(expires),
"retries": 5,
Expand Down
3 changes: 2 additions & 1 deletion automation/taskcluster/decision_task_nightly.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,8 @@
task_id=TASK_ID,
owner="android-components-team@mozilla.com",
source='{}/raw/{}/.taskcluster.yml'.format(GITHUB_HTTP_REPOSITORY, HEAD_REV),
scheduler_id=SCHEDULER_ID
scheduler_id=SCHEDULER_ID,
build_worker_type=BUILD_WORKER_TYPE,
)


Expand Down
5 changes: 3 additions & 2 deletions automation/taskcluster/lib/tasks.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,19 +8,20 @@


class TaskBuilder(object):
def __init__(self, task_id, owner, source, scheduler_id):
def __init__(self, task_id, owner, source, scheduler_id, build_worker_type):
self.task_id = task_id
self.owner = owner
self.source = source
self.scheduler_id = scheduler_id
self.build_worker_type = build_worker_type

def build_task(self, name, description, command, artifacts={}, scopes=[], features={}):
created = datetime.datetime.now()
expires = taskcluster.fromNow('1 year')
deadline = taskcluster.fromNow('1 day')

return {
"workerType": 'gecko-focus',
"workerType": self.build_worker_type,
"taskGroupId": self.task_id,
"schedulerId": self.scheduler_id,
"expires": taskcluster.stringDate(expires),
Expand Down