-
Notifications
You must be signed in to change notification settings - Fork 288
Open
Labels
blue teamdefensive programming, CI, etcdefensive programming, CI, etc
Description
Right now the re-running of benchmarks to update weights in subtensor is fairly haphazard, is not occurring on a regular basis, and is not being done on consistent / reference hardware, to alleviate this I recommend:
- Set up a custom gh runner specifically for running benchmarking jobs that matches the minimum hardware we would want for a validator. When creating the self-hosted runner, set
--max-jobs 1so that only one benchmark job can run at a time - Document the recommended/minimum validator hardware spec somewhere (if we don't already)
- Set up a CI action that runs the benchmarking suite on the custom runner. This action would be able to add a commit committing the updated weights
CI job wpossibly would be triggered as part of the tagging/release process, or manually triggered, or possibly even whenever something goes into main, pros and cons to each of those
The end goal is that subtensor's on-chain benchmarking weights always reflect the current code on consistent reference hardware without any lift from the team
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
blue teamdefensive programming, CI, etcdefensive programming, CI, etc