feat(test-utils): add CPU performance integration test harness

[sripasg]

Summary

Implements a generic CPU performance integration test harness that detects wall-clock time, CPU usage, and event loop delay regressions across key CLI scenarios, using process.cpuUsage(), monitorEventLoopDelay(), IQR outlier filtering, and ASCII chart visualization.

Details

A new top-level test directory perf-tests/ (parallel to memory-tests/) with its own vitest config. Tests run serially in a single forked process to avoid CPU contention between test samples.

• vitest.config.ts — pool: 'forks', singleFork: true, fileParallelism: false, 10-minute timeout, no retries
• globalSetup.ts — isolated $HOME, pre-downloads ripgrep binary, uses shared isolateTestEnv helper
• baselines.json — committed baseline values for all 3 scenarios
• perf-usage.test.ts — 3 scenarios:
  • cold-startup-time — measures wall-clock from process spawn to first output
  • idle-cpu-usage — monitors CPU over a 5-second idle window with event loop delay tracking
  • skill-loading-time — measures startup overhead with 20+ SKILL.md files

src/perf-test-harness.ts — PerfTestHarness class:

• measure() / measureWithEventLoop() — wraps performance.now() + process.cpuUsage() + monitorEventLoopDelay() histogram
• runScenario() — runs N samples with warmup, applies IQR outlier filtering, returns median snapshot with baseline comparison
• assertWithinBaseline() — fails if wall-clock exceeds baseline × 1.15 (15% tolerance); now also supports optional CPU usage assertion.
• updateScenarioBaseline() — writes new baseline for a scenario and logs the update.
• Refinement: Made the sample pause configurable via samplePauseMs in options.

Architectural Improvements & Fixes:

• Shared Environment Isolation — Added env-setup.ts with isolateTestEnv helper in packages/test-utils to share environment variable setup across test tiers, resolving duplication in globalSetup.ts files.
• Test Lifecycle Fix — Fixed a resource leak in perf-usage.test.ts by managing the TestRig lifecycle inside the scenario callback (using try...finally per sample).
• Test Hygiene — Replaced the TEST_ENV constant with hardcoded literals in tests to ensure they are self-contained.

Documentation and CI Updates:

• GEMINI.md — Added npm run test:perf to the Testing and Quality section.

• docs/integration-tests.md — Added a new section for Performance regression tests.

• .github/workflows/perf-nightly.yml — Created a new workflow to run performance tests nightly at 3 AM.

• npm run test:perf — compare against baselines (CI)

• npm run test:perf:update-baselines — recalibrate baselines (commit result)

• test:perf is intentionally excluded from preflight — designed for a nightly CI job, not per-PR validation

• 15% tolerance above baseline before failing; higher than memory's 10% because CPU timing is noisier in CI with shared-tenancy runners

• IQR filtering (Q1 − 1.5×IQR, Q3 + 1.5×IQR) instead of fixed thresholds to adaptively remove outlier samples

Related Issues

#24872

How to Validate

run npm run test:perf

Pre-Merge Checklist

• [✅] Updated relevant documentation and README (if needed)
• [✅] Added/updated tests (if needed)
• [NA] Noted breaking changes (if any)
• [✅] Validated on required platforms/methods:
  • [✅] MacOS
    • [✅] npm run
    • npx
    • Docker
    • Podman
    • Seatbelt
  • Windows
    • npm run
    • npx
    • Docker
  • Linux
    • npm run
    • npx
    • Docker

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request implements a robust performance regression testing framework for the CLI. By utilizing high-resolution timing primitives and statistical noise reduction techniques like IQR filtering, the new harness ensures reliable detection of performance regressions across critical CLI operations. The framework is designed for nightly CI execution to maintain performance standards without impacting per-PR preflight checks.

Highlights

• New Performance Test Harness: Introduced a generic PerfTestHarness in packages/test-utils to measure wall-clock time, CPU usage, and event loop delay using IQR outlier filtering and median sampling.
• Performance Test Suite: Added a new perf-tests/ directory with dedicated vitest configuration, baseline management, and three initial scenarios: cold startup, idle CPU usage, and skill loading time.
• CI and Documentation: Updated documentation in GEMINI.md and docs/integration-tests.md, and added a new nightly GitHub Actions workflow to run performance regression tests.

Ignored Files

• Ignored by pattern: .github/workflows/** (1)
  • .github/workflows/perf-nightly.yml

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-cli]

Hi @sripasg, thank you so much for your contribution to Gemini CLI! We really appreciate the time and effort you've put into this.

We're making some updates to our contribution process to improve how we track and review changes. Please take a moment to review our recent discussion post: Improving Our Contribution Process & Introducing New Guidelines.

Key Update: Starting January 26, 2026, the Gemini CLI project will require all pull requests to be associated with an existing issue. Any pull requests not linked to an issue by that date will be automatically closed.

Thank you for your understanding and for being a part of our community!

[github-actions]

Size Change: -4 B (0%)

Total Size: 34 MB

Filename	Size	Change
./bundle/chunk-6XX6OGDS.js	0 B	-3.16 MB (removed)	🏆
./bundle/chunk-COROWTX4.js	0 B	-14.8 MB (removed)	🏆
./bundle/core-HTSUUI7S.js	0 B	-45.5 kB (removed)	🏆
./bundle/devtoolsService-UGLGCZV3.js	0 B	-28.4 kB (removed)	🏆
./bundle/interactiveCli-KHINQZSC.js	0 B	-1.65 MB (removed)	🏆
./bundle/oauth2-provider-OELU3DIV.js	0 B	-9.16 kB (removed)	🏆
./bundle/chunk-7GGKCIOI.js	14.8 MB	+14.8 MB (new file)	🆕
./bundle/chunk-YFSDHHVF.js	3.16 MB	+3.16 MB (new file)	🆕
./bundle/core-CAQXX4VY.js	45.5 kB	+45.5 kB (new file)	🆕
./bundle/devtoolsService-ITSJWNEK.js	28.4 kB	+28.4 kB (new file)	🆕
./bundle/interactiveCli-VN5N6KML.js	1.65 MB	+1.65 MB (new file)	🆕
./bundle/oauth2-provider-PTZB5SSI.js	9.16 kB	+9.16 kB (new file)	🆕

ℹ️ View Unchanged

Filename	Size
./bundle/bundled/third_party/index.js	8 MB
./bundle/chunk-34MYV7JD.js	2.45 kB
./bundle/chunk-5AUYMPVF.js	858 B
./bundle/chunk-5PS3AYFU.js	1.18 kB
./bundle/chunk-664ZODQF.js	124 kB
./bundle/chunk-DAHVX5MI.js	206 kB
./bundle/chunk-IUUIT4SU.js	56.5 kB
./bundle/chunk-OGWWODAT.js	1.96 MB
./bundle/chunk-RJTRUG2J.js	39.8 kB
./bundle/devtools-36NN55EP.js	696 kB
./bundle/dist-T73EYRDX.js	356 B
./bundle/events-XB7DADIJ.js	418 B
./bundle/gemini.js	554 kB
./bundle/getMachineId-bsd-TXG52NKR.js	1.55 kB
./bundle/getMachineId-darwin-7OE4DDZ6.js	1.55 kB
./bundle/getMachineId-linux-SHIFKOOX.js	1.34 kB
./bundle/getMachineId-unsupported-5U5DOEYY.js	1.06 kB
./bundle/getMachineId-win-6KLLGOI4.js	1.72 kB
./bundle/memoryDiscovery-JNNGTYL3.js	980 B
./bundle/multipart-parser-KPBZEGQU.js	11.7 kB
./bundle/node_modules/@google/gemini-cli-devtools/dist/client/main.js	222 kB
./bundle/node_modules/@google/gemini-cli-devtools/dist/src/_client-assets.js	229 kB
./bundle/node_modules/@google/gemini-cli-devtools/dist/src/index.js	13.4 kB
./bundle/node_modules/@google/gemini-cli-devtools/dist/src/types.js	132 B
./bundle/sandbox-macos-permissive-open.sb	890 B
./bundle/sandbox-macos-permissive-proxied.sb	1.31 kB
./bundle/sandbox-macos-restrictive-open.sb	3.36 kB
./bundle/sandbox-macos-restrictive-proxied.sb	3.56 kB
./bundle/sandbox-macos-strict-open.sb	4.82 kB
./bundle/sandbox-macos-strict-proxied.sb	5.02 kB
./bundle/src-QVCVGIUX.js	47 kB
./bundle/tree-sitter-7U6MW5PS.js	274 kB
./bundle/tree-sitter-bash-34ZGLXVX.js	1.84 MB

_{compressed-size-action}

[gemini-code-assist]

Code Review

This pull request introduces a comprehensive CPU performance integration test harness for the Gemini CLI. It adds a new PerfTestHarness utility in packages/test-utils that measures wall-clock time, CPU usage, and event loop delays, utilizing IQR outlier filtering to ensure stable results. The PR also includes initial performance test scenarios for cold startup, idle CPU usage, and skill loading, along with necessary documentation and CI configuration. Feedback is provided regarding a resource leak in the performance tests where TestRig instances are not properly cleaned up within the sampling loop, and a reminder to use hardcoded literals in tests to align with repository style guides.

[jacob314]

Great start on the performance harness!

1. Resource Leak in Sampling Loop: In perf-usage.test.ts, the TestRig is instantiated inside the runScenario loop for every sample, but rig.cleanup() is only called in afterEach(). This means all intermediate instances from the warmups and samples leak and stay alive until the scenario completes, which will cause CPU/memory contention and heavily skew the performance results. Please move await rig.cleanup() inside the runScenario callback so each sample cleans up after itself before returning the snapshot.

2. Missing DevDependency: In perf-test-harness.ts, asciichart is dynamically imported, but it hasn't been added to package.json. Since this is just for the CLI reporting in CI, please add asciichart to devDependencies so the charts actually render in the nightly CI logs (this won't impact end-user installation size).

[jacob314]

[jacob314]