Skip to content

standardize benchmarking weights #528

@sam0x17

Description

@sam0x17

Right now the re-running of benchmarks to update weights in subtensor is fairly haphazard, is not occurring on a regular basis, and is not being done on consistent / reference hardware, to alleviate this I recommend:

  • Set up a custom gh runner specifically for running benchmarking jobs that matches the minimum hardware we would want for a validator. When creating the self-hosted runner, set --max-jobs 1 so that only one benchmark job can run at a time
  • Document the recommended/minimum validator hardware spec somewhere (if we don't already)
  • Set up a CI action that runs the benchmarking suite on the custom runner. This action would be able to add a commit committing the updated weights

CI job wpossibly would be triggered as part of the tagging/release process, or manually triggered, or possibly even whenever something goes into main, pros and cons to each of those

The end goal is that subtensor's on-chain benchmarking weights always reflect the current code on consistent reference hardware without any lift from the team

Metadata

Metadata

Assignees

Labels

blue teamdefensive programming, CI, etc

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions