[CI Problem] apache/tvm consumes too much of Apache org shared GitHub Actions resources

[lhotari]

Since September 14th, 2022, tvm has been consuming about 75% of the shared Apache org GitHub Actions resources

GitHub provides Apache with poor visibility into the usage metrics. The above report is generated with a solution developed by @TobKed, https://github.com/TobKed/fetch-apache-ga-stats (thank you @TobKed ).

When looking into the changes in twm GitHub Actions, it seems that #12695 is the source of problems. Please revert the change so that the usage gets under control.

All GitHub Actions workflow maintainers in twm project should join builds@apache.org mailing list to stay aware of possible updates in this matter.

[lhotari]

There's over 1000 workflow runs in your queue: https://github.com/apache/tvm/actions?query=is%3Aqueued and there's also 135 in progress: https://github.com/apache/tvm/actions?query=is%3Ain_progress . It seems that a lot of the ones shown in progress have been cancelled and that's probably a GitHub Actions bug ( related report community/community#31690 ).

[nicoloboschi]

You could use this script to cleanup your pending actions: https://github.com/nicoloboschi/pulsar-validation-tool/blob/master/pulsar-scripts/pulsar-gha/cancel-workflows.js

We used it in the Apache Pulsar community.
The GH bug put those runs in a weird state

run.status === "in_progress" && run.conclusion === "failure"

[potiuk]

Oh yeah. It's not nice neither for your users nor for other apache projects and seems it just went out of control for you

[potiuk]

For me it looks like your comment bot triggers itself in recursive fashion :(

[potiuk]

This one has started to heavily affect other projects (the "fairness" algorithms of GitHub likely get a very tough time with the project taking already almost 90% of the queueu @apache/tvm-committers (sorry for spam) -> i created the infra issue in the ASF asking to disable the workflow https://issues.apache.org/jira/browse/INFRA-23723 - not sure if they will/can do it but It would be great if you do it in a controlled way and cancel all the running workflow because it will start really nasty.

[gigiblender]

Thank you for raising this, and we sincerely apologize for causing this inconvenience.

#12903 should disable the bot. Regarding skipping the already queued/in-progress runs, I don't have the GH permissions to do this and will have to wait a couple of hours for someone else (possibly in a different time zone) to do it. @driazati @areusch

[potiuk]

Thanks. I see that it already started to drop slighlty - > I see that it's down from 87 % of today's workflows to 80%

[Mousius]

@gigiblender / @potiuk I've started to take care of the remaining jobs 😸

[driazati]

Thanks everyone for flagging this and disabling the job quickly!

[potiuk]

Down to 64% for today :)

[potiuk]

We are back in business :)

[potiuk]

Thanks for super-quick reaction @Mousius @gigiblender !

I also asked INFRA to see if we can work out some kind of escape hatch in case similar thing happens in the future (not necessarily with TVM :) ) in https://issues.apache.org/jira/browse/INFRA-23723

Comments/suggestions/learnings from your side would be great to add there

[potiuk]

BTW. Special kudos to @lhotari for flagging it and to @TobKed for the dashboard we have :)