Skip to content

Add mobench v0.1.37 mobile benchmark support#398

Open
dcbuild3r wants to merge 53 commits intov1from
codex/mobench-v1-browserstack
Open

Add mobench v0.1.37 mobile benchmark support#398
dcbuild3r wants to merge 53 commits intov1from
codex/mobench-v1-browserstack

Conversation

@dcbuild3r
Copy link
Copy Markdown
Collaborator

@dcbuild3r dcbuild3r commented Apr 4, 2026

Summary

  • add mobench support for ProveKit mobile benchmarking, targeting the published mobench / mobench-sdk v0.1.37 release
  • add a bench-mobile workspace crate for the monolithic passport complete_age_check flow with prepare, prove, verify, and end-to-end benchmark entry points
  • embed deterministic Noir fixtures for noir-examples/noir-passport-monolithic/complete_age_check and keep local smoke coverage for prove/verify behavior
  • add BrowserStack-backed mobile benchmark workflows for manual, label-gated, and /mobench PR-triggered runs
  • post sticky PR benchmark summaries using the <!-- mobench-summary --> marker and the mobench report/summarize output contract
  • simplify CI by deleting the bespoke validate_mobile_bench_outputs.sh script and relying on cargo-mobench ci run --fetch plus summary rendering as the output gate

Mobench version

This PR is intended to merge with mobench v0.1.37:

  • mobench = 0.1.37 for workflow installation
  • mobench-sdk = 0.1.37 for the bench-mobile crate

Device profiles

The workflows expose three profiles:

  • smoke: one Android and one iOS device for PR-triggered runs
  • triad: low, mid, and flagship Android/iOS devices for fuller manual runs
  • worst: the lowest-tier Android/iOS devices for stress-focused manual runs

Validation

Validated during PR preparation:

  • cargo metadata --no-deps --format-version 1
  • cargo check -p bench-mobile
  • actionlint .github/workflows/mobile-bench-reusable.yml
  • YAML parse check for .github/workflows/mobile-bench-reusable.yml
  • conflict-marker and whitespace checks

Not measured here

  • live BrowserStack benchmark runs were not executed from this local environment
  • sticky comment rendering and PR-triggered mobile runs depend on repository GitHub Actions plus BrowserStack secrets

@dcbuild3r dcbuild3r added codex bench Run mobile benchmarks on PRs labels Apr 4, 2026
@dcbuild3r
Copy link
Copy Markdown
Collaborator Author

/mobench platform=both iterations=30 warmup=5

@dcbuild3r dcbuild3r force-pushed the codex/mobench-v1-browserstack branch from 1da6b49 to 33c6219 Compare April 11, 2026 16:01
@dcbuild3r dcbuild3r changed the title Add mobench 0.1.28 BrowserStack benchmark pipeline Add mobench v0.1.37 mobile benchmark support Apr 27, 2026
@dcbuild3r dcbuild3r requested a review from Bisht13 April 27, 2026 20:18
@dcbuild3r dcbuild3r added bench Run mobile benchmarks on PRs and removed bench Run mobile benchmarks on PRs labels Apr 28, 2026
@dcbuild3r
Copy link
Copy Markdown
Collaborator Author

dcbuild3r commented Apr 28, 2026

Mobench Benchmark Results

Low-End BrowserStack Run

Benchmark Numbers

Target Device Function Samples Wall mean CPU total Peak growth Status
Android Vivo Y21-11.0 bench_mobile::bench_passport_complete_age_check_prove 0 n/a n/a n/a BrowserStack session timed out before testcase completion
Android Vivo Y21-11.0 bench_mobile::bench_oprf_prove 0 n/a n/a n/a Not reached in the timed-out run
Android Vivo Y21-11.0 bench_mobile::bench_p256_bigcurve_prove 0 n/a n/a n/a Not reached in the timed-out run
iOS iPhone 7-10 bench_mobile::bench_passport_complete_age_check_prove 0 n/a n/a n/a Build blocked by iOS version floor
iOS iPhone 7-10 bench_mobile::bench_oprf_prove 0 n/a n/a n/a Build blocked by iOS version floor
iOS iPhone 7-10 bench_mobile::bench_p256_bigcurve_prove 0 n/a n/a n/a Build blocked by iOS version floor

Failure Details

  • Android/Vivo Y21: BrowserStack build 285089672348d61523996b88ffde252f80689445 ended timedout. The single Vivo Y21 session timed out after 960s with TIMEDOUT: 1 and no passed/failed testcase, so no results.csv was produced.
  • iOS/iPhone 7: BrowserStack only exposes iPhone 7-10. The generated mobench iOS runner uses SwiftUI/concurrency APIs requiring iOS 13+, and Xcode 16.4 reports supported deployment targets starting at iOS 12. Lowering deployment metadata to iOS 10 causes the Swift build to fail before upload; keeping the default causes BrowserStack to reject the app/test suite as requiring a newer OS.

Implication

The current BrowserStack low-end pair is not usable for a complete 3-function mobench table with mobench 0.1.37:

  • iPhone 7-10 is below the mobench iOS runner/runtime floor.
  • Vivo Y21-11.0 is too slow for the current all-function Android smoke run under BrowserStack's 900s Espresso idle timeout.

The previous successful full benchmark table was produced on Google Pixel 7-13.0 and iPhone 16 Pro-18; this low-end rerun intentionally supersedes it with the failure state above.


Updated by Codex at 2026-04-29 16:13 UTC

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bench Run mobile benchmarks on PRs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant