Skip to content

Speedup CoT verification a bit by parallelizing network operations #786

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Eijebong merged 4 commits into from
Apr 9, 2026

Parallelize the recursion in `build_task_dependencies`

2497adc
Select commit
Loading
Failed to load commit list.
Merged

Speedup CoT verification a bit by parallelizing network operations #786

Parallelize the recursion in `build_task_dependencies`
2497adc
Select commit
Loading
Failed to load commit list.
firefoxci-taskcluster / Decision Task succeeded Apr 8, 2026 in 3m 11s

FirefoxCI (pull_request)

The task that creates all of the other tasks in the task graph

Details

View task in Taskcluster | View logs in Taskcluster | View task group in Taskcluster

Task Status

Started: 2026-04-08T17:17:48.939Z
Resolved: 2026-04-08T17:19:23.360Z
Task Execution Time: 1 minute, 34 seconds, 421 milliseconds
Task Status: completed
Reason Resolved: completed
TaskId: RbfNMs4sQEikjEI6zTf3ng
RunId: 0

Artifacts

- public/actions.json
- public/docker-contexts/python3.11.tar.gz
- public/docker-contexts/python3.12.tar.gz
- public/docker-contexts/python3.13.tar.gz
- public/docker-contexts/python3.14.tar.gz
- public/fetch-content
- public/full-task-graph.json
- public/label-to-taskid.json
- public/logs/live_backing.log
- public/logs/live.log
- public/optimizations.log
- public/parameters.yml
- public/runnable-jobs.json
- public/run-task
- public/target-tasks.json
- public/task-graph.json

[taskcluster 2026-04-08T17:17:49.002Z] Worker Type (scriptworker-1/decision) settings:
[taskcluster 2026-04-08T17:17:49.002Z]   {
[taskcluster 2026-04-08T17:17:49.002Z]     "generic-worker": {
[taskcluster 2026-04-08T17:17:49.002Z]       "config": {
[taskcluster 2026-04-08T17:17:49.002Z]         "headlessTasks": true
[taskcluster 2026-04-08T17:17:49.002Z]       },
[taskcluster 2026-04-08T17:17:49.002Z]       "engine": "multiuser",
[taskcluster 2026-04-08T17:17:49.002Z]       "go-arch": "amd64",
[taskcluster 2026-04-08T17:17:49.002Z]       "go-os": "linux",
[taskcluster 2026-04-08T17:17:49.002Z]       "go-version": "go1.26.0",
[taskcluster 2026-04-08T17:17:49.002Z]       "release": "https://github.com/taskcluster/taskcluster/releases/tag/v97.0.1",
[taskcluster 2026-04-08T17:17:49.002Z]       "revision": "6519c127aad40a47b1412136d9a5797a76eca200",
[taskcluster 2026-04-08T17:17:49.002Z]       "source": "https://github.com/taskcluster/taskcluster/commits/6519c127aad40a47b1412136d9a5797a76eca200",
[taskcluster 2026-04-08T17:17:49.002Z]       "version": "97.0.1"
[taskcluster 2026-04-08T17:17:49.002Z]     },
[taskcluster 2026-04-08T17:17:49.002Z]     "image": "projects/taskcluster-imaging/global/images/gw-fxci-gcp-l1-2404-amd64-headless-googlecompute-2026-03-05",
[taskcluster 2026-04-08T17:17:49.002Z]     "instance-id": "306087556166661003",
[taskcluster 2026-04-08T17:17:49.002Z]     "instance-type": "projects/887720501152/machineTypes/n2-standard-2",
[taskcluster 2026-04-08T17:17:49.002Z]     "local-ipv4": "10.128.0.23",
[taskcluster 2026-04-08T17:17:49.002Z]     "project-id": "fxci-production-level1-workers",
[taskcluster 2026-04-08T17:17:49.002Z]     "public-hostname": "scriptworker-1-decision-udj-1n96rw2lhjizv9mjeg.c.fxci-production-level1-workers.internal",
[taskcluster 2026-04-08T17:17:49.002Z]     "public-ipv4": "35.193.210.214",
[taskcluster 2026-04-08T17:17:49.002Z]     "region": "us-central1",
[taskcluster 2026-04-08T17:17:49.002Z]     "zone": "us-central1-b"
[taskcluster 2026-04-08T17:17:49.002Z]   }
[taskcluster 2026-04-08T17:17:49.002Z] Task ID: RbfNMs4sQEikjEI6zTf3ng
[taskcluster 2026-04-08T17:17:49.002Z] === Task Starting ===
[taskcluster 2026-04-08T17:17:50.967Z] [d2g] Loading docker image
[taskcluster 2026-04-08T17:18:05.370Z] [d2g] Loaded docker image "docker.io/mozillareleases/taskgraph:decision-v20.0.0@sha256:afae5e5d3caaeb10a93fae56bc3176d1e73faf9b616c01facf811e488db4eafe"
[taskcluster 2026-04-08T17:18:05.371Z] Executing command 0: docker run -t --name taskcontainer_LzMvLm_3TUqxcpfWjUMg-w --memory-swap -1 --pids-limit -1 '--add-host=localhost.localdomain:127.0.0.1' --add-host=taskcluster:host-gateway --env-file 'env.list' 'mozillareleases/taskgraph:decision-v20.0.0@sha256:afae5e5d3caaeb10a93fae56bc3176d1e73faf9b616c01facf811e488db4eafe' run-task --scriptworker-checkout=/builds/worker/checkouts/src -- bash -cx 'cd /builds/worker/checkouts/src && ln -s /builds/worker/artifacts artifacts && taskgraph decision --pushlog-id='\''0'\'' --pushdate='\''0'\'' --project='\''scriptworker'\'' --owner='\''Eijebong@users.noreply.github.com'\'' --level='\''1'\'' --repository-type=git --tasks-for='\''github-pull-request'\'' --base-repository='\''https://github.com/mozilla-releng/scriptworker'\'' --base-ref='\''main'\'' --base-rev='\''6b0b66b03e9ec07b462a550b19b742a863d6b17a'\'' --head-repository='\''https://github.com/Eijebong/scriptworker'\'' --head-ref='\''speedup-cot-verification'\'' --head-rev='\''2497adcbfed4dc5f3b48ae8944ed1957f4699a83'\'' 
[taskcluster 2026-04-08T17:18:05.371Z] '
[setup 2026-04-08T17:18:07.793+00:00] run-task started in /
[setup 2026-04-08T17:18:07.793+00:00] Invoked by command: --scriptworker-checkout=/builds/worker/checkouts/src -- bash -cx cd /builds/worker/checkouts/src && ln -s /builds/worker/artifacts artifacts && taskgraph decision --pushlog-id='0' --pushdate='0' --project='scriptworker' --owner='Eijebong@users.noreply.github.com' --level='1' --repository-type=git --tasks-for='github-pull-request' --base-repository='https://github.com/mozilla-releng/scriptworker' --base-ref='main' --base-rev='6b0b66b03e9ec07b462a550b19b742a863d6b17a' --head-repository='https://github.com/Eijebong/scriptworker' --head-ref='speedup-cot-verification' --head-rev='2497adcbfed4dc5f3b48ae8944ed1957f4699a83' 

[setup 2026-04-08T17:18:07.794+00:00] Python version: 3.11.2
[setup 2026-04-08T17:18:07.795+00:00] Subprocess python version: 
cpython-3.11.2-linux-x86_64-gnu    /usr/bin/python3.11
cpython-3.11.2-linux-x86_64-gnu    /usr/bin/python3 -> python3.11
[vcs 2026-04-08T17:18:07.943+00:00] WARNING: vcs checkout path (/builds/worker/checkouts/src) not in cache or volume; performance will likely suffer
[setup 2026-04-08T17:18:07.945+00:00] running as worker:worker
[vcs 2026-04-08T17:18:07.945+00:00] executing ['git', 'config', '--global', '--add', 'safe.directory', '/builds/worker/checkouts/src']
[vcs 2026-04-08T17:18:07.950+00:00] executing ['git', 'clone', 'https://github.com/mozilla-releng/scriptworker', '/builds/worker/checkouts/src']
[vcs 2026-04-08T17:18:07.969+00:00] Cloning into '/builds/worker/checkouts/src'...
[vcs 2026-04-08T17:18:09.477+00:00] executing ['git', 'fetch', 'origin', '6b0b66b03e9ec07b462a550b19b742a863d6b17a']
[vcs 2026-04-08T17:18:09.779+00:00] From https://github.com/mozilla-releng/scriptworker
[vcs 2026-04-08T17:18:09.779+00:00]  * branch            6b0b66b03e9ec07b462a550b19b742a863d6b17a -> FETCH_HEAD
[vcs 2026-04-08T17:18:09.787+00:00] executing ['git', 'fetch', 'https://github.com/Eijebong/scriptworker', 'speedup-cot-verification']
[vcs 2026-04-08T17:19:18.452+00:00] From https://github.com/Eijebong/scriptworker
[vcs 2026-04-08T17:19:18.453+00:00]  * branch            speedup-cot-verification -> FETCH_HEAD
[vcs 2026-04-08T17:19:18.455+00:00] executing ['git', 'checkout', '-f', '-B', 'speedup-cot-verification', '2497adcbfed4dc5f3b48ae8944ed1957f4699a83']
[vcs 2026-04-08T17:19:18.604+00:00] Switched to a new branch 'speedup-cot-verification'
[vcs 2026-04-08T17:19:18.605+00:00] cleaning git checkout...
[vcs 2026-04-08T17:19:18.605+00:00] executing ['git', 'clean', '-nxdff']
[vcs 2026-04-08T17:19:18.608+00:00] removing []
[vcs 2026-04-08T17:19:18.608+00:00] successfully cleaned git checkout!
[vcs 2026-04-08T17:19:18.610+00:00] TinderboxPrint:<a href='https://github.com/Eijebong/scriptworker/commit/2497adcbfed4dc5f3b48ae8944ed1957f4699a83' title='Built from scriptworker commit 2497adcbfed4dc5f3b48ae8944ed1957f4699a83'>2497adcbfed4dc5f3b48ae8944ed1957f4699a83</a>
[task 2026-04-08T17:19:18.610+00:00] executing ['bash', '-cx', "cd /builds/worker/checkouts/src && ln -s /builds/worker/artifacts artifacts && taskgraph decision --pushlog-id='0' --pushdate='0' --project='scriptworker' --owner='Eijebong@users.noreply.github.com' --level='1' --repository-type=git --tasks-for='github-pull-request' --base-repository='https://github.com/mozilla-releng/scriptworker' --base-ref='main' --base-rev='6b0b66b03e9ec07b462a550b19b742a863d6b17a' --head-repository='https://github.com/Eijebong/scriptworker' --head-ref='speedup-cot-verification' --head-rev='2497adcbfed4dc5f3b48ae8944ed1957f4699a83' \n"]
[task 2026-04-08T17:19:18.612+00:00] + cd /builds/worker/checkouts/src
[task 2026-04-08T17:19:18.612+00:00] + ln -s /builds/worker/artifacts artifacts
[task 2026-04-08T17:19:18.614+00:00] + taskgraph decision --pushlog-id=0 --pushdate=0 --project=scriptworker --owner=Eijebong@users.noreply.github.com --level=1 --repository-type=git --tasks-for=github-pull-request --base-repository=https://github.com/mozilla-releng/scriptworker --base-ref=main --base-rev=6b0b66b03e9ec07b462a550b19b742a863d6b17a --head-repository=https://github.com/Eijebong/scriptworker --head-ref=speedup-cot-verification --head-rev=2497adcbfed4dc5f3b48ae8944ed1957f4699a83
[task 2026-04-08T17:19:19.623+00:00] 2026-04-08 17:19:19,623 - INFO - Loading graph configuration.
[task 2026-04-08T17:19:19.635+00:00] 2026-04-08 17:19:19,635 - WARNING - using default project parameters; add scriptworker to PER_PROJECT_PARAMETERS in /usr/local/lib/python3.11/dist-packages/taskgraph/decision.py to customize behavior for this project
[task 2026-04-08T17:19:19.637+00:00] 2026-04-08 17:19:19,637 - INFO - Using Parameters(id=4743d02f44d0) (from defaults)
[task 2026-04-08T17:19:19.638+00:00] 2026-04-08 17:19:19,637 - INFO - writing artifact file `parameters.yml`
[task 2026-04-08T17:19:19.653+00:00] 2026-04-08 17:19:19,652 - INFO - writing artifact file `actions.json`
[task 2026-04-08T17:19:19.654+00:00] 2026-04-08 17:19:19,654 - INFO - Loading kinds
[task 2026-04-08T17:19:19.657+00:00] 2026-04-08 17:19:19,657 - INFO - Generating full task set
[task 2026-04-08T17:19:19.685+00:00] 2026-04-08 17:19:19,684 - INFO - Writing docker-contexts/python3.14.tar.gz for docker image python3.14
[task 2026-04-08T17:19:19.707+00:00] 2026-04-08 17:19:19,707 - INFO - Writing docker-contexts/python3.13.tar.gz for docker image python3.13
[task 2026-04-08T17:19:19.728+00:00] 2026-04-08 17:19:19,727 - INFO - Writing docker-contexts/python3.12.tar.gz for docker image python3.12
[task 2026-04-08T17:19:19.748+00:00] 2026-04-08 17:19:19,748 - INFO - Writing docker-contexts/python3.11.tar.gz for docker image python3.11
[task 2026-04-08T17:19:19.771+00:00] 2026-04-08 17:19:19,771 - INFO - Generated 4 tasks for kind docker-image
[task 2026-04-08T17:19:19.817+00:00] 2026-04-08 17:19:19,817 - INFO - Generated 9 tasks for kind tox
[task 2026-04-08T17:19:19.822+00:00] 2026-04-08 17:19:19,822 - INFO - Generated 1 tasks for kind pr
[task 2026-04-08T17:19:19.829+00:00] 2026-04-08 17:19:19,828 - INFO - Generating full task graph
[task 2026-04-08T17:19:19.829+00:00] 2026-04-08 17:19:19,828 - INFO - Full task graph contains 14 tasks and 9 dependencies
[task 2026-04-08T17:19:19.831+00:00] 2026-04-08 17:19:19,830 - INFO - writing artifact file `full-task-graph.json`
[task 2026-04-08T17:19:19.833+00:00] 2026-04-08 17:19:19,832 - INFO - writing artifact file `runnable-jobs.json`
[task 2026-04-08T17:19:19.833+00:00] 2026-04-08 17:19:19,833 - INFO - Generating target task set
[task 2026-04-08T17:19:19.834+00:00] 2026-04-08 17:19:19,834 - INFO - Filter filter_target_tasks pruned 4 tasks (10 remain)
[task 2026-04-08T17:19:19.834+00:00] 2026-04-08 17:19:19,834 - INFO - writing artifact file `target-tasks.json`
[task 2026-04-08T17:19:19.835+00:00] 2026-04-08 17:19:19,835 - INFO - Generating target task graph
[task 2026-04-08T17:19:19.836+00:00] 2026-04-08 17:19:19,835 - INFO - Adding 4 tasks with `always_target` attribute
[task 2026-04-08T17:19:19.836+00:00] 2026-04-08 17:19:19,836 - INFO - Generating optimized task graph
[task 2026-04-08T17:19:19.838+00:00] 2026-04-08 17:19:19,837 - INFO - No tasks removed during optimization
[task 2026-04-08T17:19:20.373+00:00] 2026-04-08 17:19:20,373 - INFO - Replaced 4 tasks by index-search (index-search) during optimization.
[task 2026-04-08T17:19:20.375+00:00] 2026-04-08 17:19:20,375 - INFO - writing artifact file `task-graph.json`
[task 2026-04-08T17:19:20.376+00:00] 2026-04-08 17:19:20,376 - INFO - writing artifact file `label-to-taskid.json`
[task 2026-04-08T17:19:20.379+00:00] 2026-04-08 17:19:20,379 - INFO - Creating task with taskId fPjDWgSrSlCn2XWG-HyVug for tox-py311
[task 2026-04-08T17:19:20.381+00:00] 2026-04-08 17:19:20,380 - INFO - Creating task with taskId Bwvdg9GvQjOOJ25-CrB5tw for tox-check
[task 2026-04-08T17:19:20.387+00:00] 2026-04-08 17:19:20,387 - INFO - Creating task with taskId LBm1USkvSaCkWHY6Vado_Q for tox-py314-cot
[task 2026-04-08T17:19:20.391+00:00] 2026-04-08 17:19:20,391 - INFO - Creating task with taskId PW1enjkMQemJE1DkRBaxIQ for tox-py313
[task 2026-04-08T17:19:20.394+00:00] 2026-04-08 17:19:20,394 - INFO - Creating task with taskId KKNuX7hlQiiiBrIxPN2hkQ for tox-py313-cot
[task 2026-04-08T17:19:20.399+00:00] 2026-04-08 17:19:20,398 - INFO - Creating task with taskId UeEUGJpGSzqfojLR9CvZgA for tox-py312
[task 2026-04-08T17:19:20.402+00:00] 2026-04-08 17:19:20,402 - INFO - Creating task with taskId V7rXqCPwQDSGF9fjUEsEHg for tox-py311-cot
[task 2026-04-08T17:19:20.406+00:00] 2026-04-08 17:19:20,406 - INFO - Creating task with taskId WpfQ-QNzRQqjkJjxJrNMDg for tox-py314
[task 2026-04-08T17:19:20.410+00:00] 2026-04-08 17:19:20,410 - INFO - Creating task with taskId DKciMcqgRVuo2O6tgvCUUA for tox-py312-cot
[task 2026-04-08T17:19:20.875+00:00] 2026-04-08 17:19:20,874 - INFO - Creating task with taskId flGSzqiTTUavJsrU8pad6A for pr-complete
[taskcluster 2026-04-08T17:19:21.777Z]                        Exit Code: 0
[taskcluster 2026-04-08T17:19:21.777Z]                        User Time: 18.568ms
[taskcluster 2026-04-08T17:19:21.777Z]                      Kernel Time: 19.545ms
[taskcluster 2026-04-08T17:19:21.777Z]                        Wall Time: 1m16.403488472s
[taskcluster 2026-04-08T17:19:21.777Z]  Average Available System Memory: 6.93 GiB
[taskcluster 2026-04-08T17:19:21.777Z]       Average System Memory Used: 837.61 MiB
[taskcluster 2026-04-08T17:19:21.777Z]          Peak System Memory Used: 1.05 GiB
[taskcluster 2026-04-08T17:19:21.777Z]              Total System Memory: 7.75 GiB
[taskcluster 2026-04-08T17:19:21.777Z]                           Result: SUCCEEDED
[taskcluster 2026-04-08T17:19:21.777Z] === Task Finished ===
[taskcluster 2026-04-08T17:19:21.777Z] Task Duration: 1m16.406269425s
View more details on firefoxci-taskcluster