Build automated benchmarking infrastructure to measure and track model performance across devices and configurations.
Currently, benchmarking models consumes a lot of time:
- Manual testing on each device
- No standardized benchmark suite
- Results scattered across different runs/people
- No comparison between model configurations (quantization, delegates)
Goals
- Automated benchmark suite
- Standardized metrics across all models
- Historical tracking to catch performance regressions
Build automated benchmarking infrastructure to measure and track model performance across devices and configurations.
Currently, benchmarking models consumes a lot of time:
Goals