Skip to content

[ci][tvm-bot] Re-running an in-progress github action workflow fails #12847

@driazati

Description

@driazati

See #12785 (comment) on #12785:

This happened while one job was failed (MacOS) and 2 were still running (Android and Windows). The rerun request went to workflow 3091384727 which was the stopped MacOS build, but it might have errored out since the MacOS is a failed job within a workflow that contains other running jobs (Windows / MacOS). The fix here might be to break up main.yml in .github/workflows to macos.yml / windows.yml / android.yml (though the rerun-failed-jobs endpoint sounds like it should work fine with this case). We could also try cancelling the whole workflow, waiting a bit, then issuing the rerun.

Traceback (most recent call last):
  File "/home/runner/work/tvm/tvm/ci/scripts/git_utils.py", line 121, in _request
    with request.urlopen(req, data) as response:
  File "/usr/lib/python3.8/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/lib/python3.8/urllib/request.py", line 640, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.8/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "ci/scripts/github_tvmbot.py", line 574, in comment_failure
    raise item
  File "ci/scripts/github_tvmbot.py", line 685, in run
    pr.rerun_github_actions()
  File "ci/scripts/github_tvmbot.py", line 563, in rerun_github_actions
    raise e
  File "ci/scripts/github_tvmbot.py", line 555, in rerun_github_actions
    actions_github.post(f"actions/runs/{workflow_id}/rerun-failed-jobs", data={})
  File "/home/runner/work/tvm/tvm/ci/scripts/git_utils.py", line 143, in post
    return self._request(self.base + url, data, method="POST")
  File "/home/runner/work/tvm/tvm/ci/scripts/git_utils.py", line 126, in _request
    raise RuntimeError(f"Error response: {msg}\n{error_data}")
RuntimeError: Error response: HTTP Error 403: Forbidden
{"message":"This workflow is already running","documentation_url":"https://docs.github.com/rest/reference/actions#re-run-workflow-failed-jobs"}

cc @Mousius @areusch @gigiblender

Metadata

Metadata

Assignees

No one assigned

    Labels

    dev:cineeds-triagePRs or issues that need to be investigated by maintainers to find the right assignees to address it

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions