Generalize evals infra to support more types of evals, organization and queuing of named suites by gundermanc · Pull Request #24941 · google-gemini/gemini-cli

gundermanc · 2026-04-08T17:42:47Z

Generalizes the evals infrastructure to:

Support grouping of evals into named 'suites' which can be run by name.
Support categorization of evals into behavioral (what we have today), hero (small number of larger and more representative evals), and component level (benchmarks for a specific component's effectiveness).
Helpers for component level evals.

This is a precursor to having component level evaluation suites, such as one for compression, which measures the effectiveness of that component in isolation.

It also paves the way for having more and varied tests.

Only the behavioral evals tests are run by default. Other suites can be run on demand by feature maintainers or autonomously at PR time via #24877

Here's what the run queue UI looks like:

github-actions · 2026-04-08T17:48:44Z

Size Change: -4 B (0%)

Total Size: 34 MB

Filename	Size	Change
`./bundle/chunk-BR7HWKM6.js`	0 B	-14.8 MB (removed)	🏆
`./bundle/chunk-G2OX3PJV.js`	0 B	-3.16 MB (removed)	🏆
`./bundle/core-7VWIR6XH.js`	0 B	-45.5 kB (removed)	🏆
`./bundle/devtoolsService-RL2REABW.js`	0 B	-28.4 kB (removed)	🏆
`./bundle/interactiveCli-L2LDGJEF.js`	0 B	-1.65 MB (removed)	🏆
`./bundle/oauth2-provider-3RJIPEOL.js`	0 B	-9.16 kB (removed)	🏆
`./bundle/chunk-FLBCTWLQ.js`	14.8 MB	+14.8 MB (new file)	🆕
`./bundle/chunk-GQYCADIT.js`	3.16 MB	+3.16 MB (new file)	🆕
`./bundle/core-STKXOIN5.js`	45.5 kB	+45.5 kB (new file)	🆕
`./bundle/devtoolsService-Z2X6MK2F.js`	28.4 kB	+28.4 kB (new file)	🆕
`./bundle/interactiveCli-XG7HS3CB.js`	1.65 MB	+1.65 MB (new file)	🆕
`./bundle/oauth2-provider-IR3USCIC.js`	9.16 kB	+9.16 kB (new file)	🆕

ℹ️ View Unchanged

Filename	Size
`./bundle/bundled/third_party/index.js`	8 MB
`./bundle/chunk-34MYV7JD.js`	2.45 kB
`./bundle/chunk-5AUYMPVF.js`	858 B
`./bundle/chunk-5PS3AYFU.js`	1.18 kB
`./bundle/chunk-664ZODQF.js`	124 kB
`./bundle/chunk-DAHVX5MI.js`	206 kB
`./bundle/chunk-IUUIT4SU.js`	56.5 kB
`./bundle/chunk-OGWWODAT.js`	1.96 MB
`./bundle/chunk-RJTRUG2J.js`	39.8 kB
`./bundle/devtools-36NN55EP.js`	696 kB
`./bundle/dist-T73EYRDX.js`	356 B
`./bundle/events-XB7DADIJ.js`	418 B
`./bundle/gemini.js`	554 kB
`./bundle/getMachineId-bsd-TXG52NKR.js`	1.55 kB
`./bundle/getMachineId-darwin-7OE4DDZ6.js`	1.55 kB
`./bundle/getMachineId-linux-SHIFKOOX.js`	1.34 kB
`./bundle/getMachineId-unsupported-5U5DOEYY.js`	1.06 kB
`./bundle/getMachineId-win-6KLLGOI4.js`	1.72 kB
`./bundle/memoryDiscovery-JNNGTYL3.js`	980 B
`./bundle/multipart-parser-KPBZEGQU.js`	11.7 kB
`./bundle/node_modules/@google/gemini-cli-devtools/dist/client/main.js`	222 kB
`./bundle/node_modules/@google/gemini-cli-devtools/dist/src/_client-assets.js`	229 kB
`./bundle/node_modules/@google/gemini-cli-devtools/dist/src/index.js`	13.4 kB
`./bundle/node_modules/@google/gemini-cli-devtools/dist/src/types.js`	132 B
`./bundle/sandbox-macos-permissive-open.sb`	890 B
`./bundle/sandbox-macos-permissive-proxied.sb`	1.31 kB
`./bundle/sandbox-macos-restrictive-open.sb`	3.36 kB
`./bundle/sandbox-macos-restrictive-proxied.sb`	3.56 kB
`./bundle/sandbox-macos-strict-open.sb`	4.82 kB
`./bundle/sandbox-macos-strict-proxied.sb`	5.02 kB
`./bundle/src-QVCVGIUX.js`	47 kB
`./bundle/tree-sitter-7U6MW5PS.js`	274 kB
`./bundle/tree-sitter-bash-34ZGLXVX.js`	1.84 MB

_{compressed-size-action}

gemini-code-assist · 2026-04-08T22:08:32Z

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request generalizes the existing evaluation infrastructure to support more flexible testing scenarios. By introducing suite-based grouping and categorization, it enables more granular control over which tests are executed. Furthermore, it adds infrastructure for component-level evaluations and improves test reliability by implementing a retry mechanism for API-related failures, ensuring a more stable CI/CD pipeline.

Highlights

Evaluation Infrastructure Generalization: Introduced support for grouping evaluations into named 'suites' and categorizing them into behavioral, hero, and component-level types.
Component-Level Evaluation Support: Added a new component-test-helper.ts to enable running evaluations directly against backend components, bypassing the UI.
Reliability Improvements: Implemented withEvalRetries to handle transient API errors gracefully, preventing flaky tests from blocking PRs.
Configuration Updates: Updated existing evaluation test cases to include the new suiteName and suiteType metadata.

Ignored Files

Ignored by pattern: .github/workflows/** (2)
- .github/workflows/chained_e2e.yml
- .github/workflows/evals-nightly.yml

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request refactors the evaluation framework to introduce test suites and improve execution reliability. Key updates include adding suiteName and suiteType metadata to all evaluations, implementing a withEvalRetries helper to manage transient API errors, and adding a componentEvalTest utility for direct backend component testing. Feedback identifies a restricted relative import in component-test-helper.ts that violates repository conventions, as well as issues in vitest.config.ts where the core package alias path appears incorrect and the necessary React alias was removed.

alisa-alisa · 2026-04-08T22:53:25Z

+    suiteName: 'default',
+    suiteType: 'behavioral',


Are these fields mandatory? Meaning can we ensure that:

Developers can't forget them

Developers can't misspell them (e.g. "behavioral" and "behavioural" etc) and we always have known limited names

What about intersecting suites? Can it be that the test might be needed for 2 validations or is it not the case at all?

Yes, they're mandatory. The suiteType is strongly typed. It has to be one of 3 well known strings.

What about intersecting suites? Can it be that the test might be needed for 2 validations or is it not the case at all?

Not yet supported but I think that's a reasonable thing to add in a follow up. For right now the goal is to enable grouping, primarily so that we have a way to run a specific subset of tests manually.

I think it's ok to make whatever adjustments are needed to make the related tests stuff work.

I would suggest to have any sort of enum for the suites names and suiteType. Strings are not designed to be mandated to be spelled correctly.

It's not typed as a string, it's typed as a discriminated union 'behavioral' | 'hero-scenario' | 'component-eval'. The TS compiler generates an error if it's a string other than one of those 3.

alisa-alisa

Approved with comment

alisa-alisa · 2026-04-08T22:59:53Z

+    suiteName: 'default',
+    suiteType: 'behavioral',


I would suggest to have any sort of enum for the suites names and suiteType. Strings are not designed to be mandated to be spelled correctly.

scidomino · 2026-04-08T23:08:12Z

+  makeFakeConfig,
+  type GeminiCLIExtension,
+} from '@google/gemini-cli-core';
+import { createMockSettings } from '../packages/cli/src/test-utils/settings.js';


Is this import ok?

It seems to be ok for now. I think longer term we need to refactor these helpers into cleaner package boundaries.

github-actions · 2026-04-08T23:56:04Z

✅ 66 tests passed successfully on gemini-3-flash-preview.

This is an automated guidance message triggered by steering logic signatures.

…nd queuing of named suites (google-gemini#24941)

Blind-blind winner: opus Complexity gate: delta=0 Reviewer: approved

* refactor(plan): simplify policy priorities and consolidate read-only rules (google-gemini#24849) * feat(test-utils): add memory usage integration test harness (google-gemini#24876) * feat(memory): add /memory inbox command for reviewing extracted skills (google-gemini#24544) * chore(release): bump version to 0.39.0-nightly.20260408.e77b22e63 (google-gemini#24939) * fix(core): ensure robust sandbox cleanup in all process execution paths (google-gemini#24763) Co-authored-by: Spencer <spencertang@google.com> * chore: update ink version to 6.6.8 (google-gemini#24934) * Changelog for v0.38.0-preview.0 (google-gemini#24938) Co-authored-by: gemini-cli-robot <224641728+gemini-cli-robot@users.noreply.github.com> Co-authored-by: g-samroberts <samroberts@google.com> * chore: ignore conductor directory (google-gemini#22128) Co-authored-by: Coco Sheng <cocosheng@google.com> * Changelog for v0.37.0 (google-gemini#24940) Co-authored-by: gemini-cli-robot <224641728+gemini-cli-robot@users.noreply.github.com> Co-authored-by: Sam Roberts <158088236+g-samroberts@users.noreply.github.com> * feat(plan): require user confirmation for activate_skill in Plan Mode (google-gemini#24946) * feat(test-utils): add CPU performance integration test harness (google-gemini#24951) * fix(core): resolve windows symlink bypass and stabilize sandbox integration tests (google-gemini#24834) * test(sdk): add unit tests for GeminiCliSession (google-gemini#21897) * fix(cli): restore file path display in edit and write tool confirmations (google-gemini#24974) * fix(cli-ui): enable Ctrl+Backspace for word deletion in Windows Terminal (google-gemini#21447) * fix(core): dynamic session ID injection to resolve resume bugs (google-gemini#24972) * Update ink version to 6.6.9 (google-gemini#24980) * feat(core): refine shell tool description display logic (google-gemini#24903) * Generalize evals infra to support more types of evals, organization and queuing of named suites (google-gemini#24941) * fix(cli): optimize startup with lightweight parent process (google-gemini#24667) * refactor(sandbox): use centralized sandbox paths in macOS Seatbelt implementation (google-gemini#24984) * feat(cli): refine tool output formatting for compact mode (google-gemini#24677) * fix(sdk): skip broken sendStream tests to unblock nightly (google-gemini#25000) * refactor(core): use centralized path resolution for Linux sandbox (google-gemini#24985) * Support ctrl+shift+g (google-gemini#25035) * feat(core): refactor subagent tool to unified invoke_subagent tool (google-gemini#24489) * fix(core): add explicit git identity env vars to prevent sandbox checkpointing error (google-gemini#19775) Co-authored-by: David Pierce <davidapierce@google.com> * fix: respect hideContextPercentage when FooterConfigDialog is closed without changes (google-gemini#24773) Co-authored-by: Coco Sheng <cocosheng@google.com> * fix(cli): suppress unhandled AbortError logs during request cancellation (google-gemini#22621) * Automated documentation audit (google-gemini#24567) * feat(cli): implement useAgentStream hook (google-gemini#24292) Co-authored-by: Adam Weidman <adamfweidman@gmail.com> Co-authored-by: Adam Weidman <adamfweidman@google.com> * refactor(core): remove legacy subagent wrapping tools (google-gemini#25053) * refactor(plan) Clean default plan toml (google-gemini#25037) * fix(core): honor retryDelay in RetryInfo for 503 errors (google-gemini#25057) * fix(core): remediate subagent memory leaks using AbortSignal in MessageBus (google-gemini#25048) * feat(cli): wire up useAgentStream in AppContainer (google-gemini#24297) Co-authored-by: Adam Weidman <adamfweidman@gmail.com> Co-authored-by: Adam Weidman <adamfweidman@google.com> * feat(core): migrate chat recording to JSONL streaming (google-gemini#23749) * fix(core): clear 5-minute timeouts in oauth flow to prevent memory leaks (google-gemini#24968) * fix(sandbox): centralize async git worktree resolution and enforce read-only security (google-gemini#25040) * feat(test): add high-volume shell test and refine perf harness (google-gemini#24983) * fix(core): silently handle EPERM when listing dir structure (google-gemini#25066) * Changelog for v0.37.1 (google-gemini#25055) Co-authored-by: gemini-cli-robot <224641728+gemini-cli-robot@users.noreply.github.com> * fix: decode Uint8Array and multi-byte UTF-8 in API error messages (google-gemini#23341) Co-authored-by: Coco Sheng <cocosheng@google.com> * Automated documentation audit results (google-gemini#22755) * debugging(ui): add optional debugRainbow setting (google-gemini#25088) * fix: resolve lifecycle memory leaks by cleaning up listeners and root closures (google-gemini#25049) * docs(cli): updates f12 description to be more precise (google-gemini#15816) * fix(cli): mark /settings as unsafe to run concurrently (google-gemini#25061) * fix(core): remove buffer slice to prevent OOM on large output streams (google-gemini#25094) * feat(core): persist subagent agentId in tool call records (google-gemini#25092) * chore(core): increase codebase investigator turn limits to 50 (google-gemini#25125) * refactor(core): consolidate execute() arguments into ExecuteOptions (google-gemini#25101) * feat(core): add Strategic Re-evaluation guidance to system prompt (google-gemini#25062) * fix(core): preserve shell execution config fields on update (google-gemini#25113) * docs: add vi shortcuts and clarify MCP sandbox setup (google-gemini#21679) Co-authored-by: Jenna Inouye <jinouye@google.com> * fix(cli): pass session id to interactive shell executions (google-gemini#25114) * fix(cli): resolve text sanitization data loss due to C1 control characters (google-gemini#22624) * feat(core): add large memory regression test (google-gemini#25059) * fix(core): resolve PTY exhaustion and orphan MCP subprocess leaks (google-gemini#25079) * chore: switch from keytar to @github/keytar (google-gemini#25143) * chore(deps): update vulnerable dependencies via npm audit fix (google-gemini#25140) * perf(sandbox): optimize Windows sandbox initialization via native ACL application (google-gemini#25077) * fix: improve audio MIME normalization and validation in file reads (google-gemini#21636) Co-authored-by: Coco Sheng <cocosheng@google.com> * docs: Update docs-audit to include changes in PR body (google-gemini#25153) * docs: correct documentation for enforced authentication type (google-gemini#25142) * fix(cli): exclude update_topic from confirmation queue count (google-gemini#24945) * Memory fix for trace's streamWrapper. (google-gemini#25089) * fix(core): fix quota footer for non-auto models and improve display (google-gemini#25121) * docs(contributing): clarify self-assignment policy for issues (google-gemini#23087) * feat(core): add skill patching support with /memory inbox integration (google-gemini#25148) * Stop suppressing thoughts and text in model response (google-gemini#25073) * fix(release): prefix git hash in nightly versions to prevent semver normalization (google-gemini#25304) * feat(cli): extract QuotaContext and resolve infinite render loop (google-gemini#24959) * refactor(core): extract and centralize sandbox path utilities (google-gemini#25305) Co-authored-by: David Pierce <davidapierce@google.com> * feat(ui): added enhancements to scroll momentum (google-gemini#24447) * fix(core): replace custom binary detection with isbinaryfile to correctly handle UTF-8 (U+FFFD) (google-gemini#25297) * feat(agent): implement tool-controlled display protocol (Steps 2-3) (google-gemini#25134) * Stop showing scrollbar unless we are in terminalBuffer mode (google-gemini#25320) * fix(core): expose GEMINI_PLANS_DIR to hook environment (google-gemini#25296) Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * feat: support auth block in MCP servers config in agents (google-gemini#24770) * feat(core): implement silent fallback for Plan Mode model routing (google-gemini#25317) * fix: correct redirect count increment in fetchJson (google-gemini#24896) Co-authored-by: Tommaso Sciortino <sciortino@gmail.com> * fix(core): prevent secondary crash in ModelRouterService finally block (google-gemini#25333) * feat(core): introduce decoupled ContextManager and Sidecar architecture (google-gemini#24752) * docs(core): update generalist agent documentation (google-gemini#25325) * chore(mcp): check MCP error code over brittle string match (google-gemini#25381) * test(core): improve sandbox integration test coverage and fix OS-specific failures (google-gemini#25307) Co-authored-by: David Pierce <davidapierce@google.com> * feat(plan): update plan mode prompt to allow showing plan content (google-gemini#25058) * fix(core): use debug level for keychain fallback logging (google-gemini#25398) * feat(test): add a performance test in asian language (google-gemini#25392) * feat(cli): enable mouse clicking for cursor positioning in AskUser multi-line answers (google-gemini#24630) * fix(core): detect kmscon terminal as supporting true color (google-gemini#25282) Co-authored-by: Adib234 <30782825+Adib234@users.noreply.github.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * ci: add agent session drift check workflow (google-gemini#25389) * use macos-latest-large runner where applicable. (google-gemini#25413) * Changelog for v0.37.2 (google-gemini#25336) Co-authored-by: gemini-cli-robot <224641728+gemini-cli-robot@users.noreply.github.com> * chore(release): v0.39.0-preview.0 * fix(patch): cherry-pick a4e98c0 to release/v0.39.0-preview.0-pr-25138 to patch version v0.39.0-preview.0 and create version 0.39.0-preview.1 (google-gemini#25766) Co-authored-by: Mahima Shanware <mahima.shanware@gmail.com> * chore(release): v0.39.0-preview.1 * fix(patch): cherry-pick d6f88f8 to release/v0.39.0-preview.1-pr-25670 to patch version v0.39.0-preview.1 and create version 0.39.0-preview.2 (google-gemini#25776) Co-authored-by: Adam Weidman <65992621+adamfweidman@users.noreply.github.com> * chore(release): v0.39.0-preview.2 * chore(release): v0.39.0 --------- Co-authored-by: ruomeng <ruomeng@google.com> Co-authored-by: Sri Pasumarthi <111310667+sripasg@users.noreply.github.com> Co-authored-by: Sandy Tao <sandytao520@icloud.com> Co-authored-by: gemini-cli-robot <gemini-cli-robot@google.com> Co-authored-by: Emily Hedlund <ehedlund@google.com> Co-authored-by: Spencer <spencertang@google.com> Co-authored-by: Jacob Richman <jacob314@gmail.com> Co-authored-by: gemini-cli-robot <224641728+gemini-cli-robot@users.noreply.github.com> Co-authored-by: g-samroberts <samroberts@google.com> Co-authored-by: JAYADITYA <96861162+JayadityaGit@users.noreply.github.com> Co-authored-by: Coco Sheng <cocosheng@google.com> Co-authored-by: Sam Roberts <158088236+g-samroberts@users.noreply.github.com> Co-authored-by: Adamya Singh <adamyasingh54@gmail.com> Co-authored-by: Jarrod Whelan <150866123+jwhelangoog@users.noreply.github.com> Co-authored-by: dogukanozen <dogukannozen@hotmail.com> Co-authored-by: Tommaso Sciortino <sciortino@gmail.com> Co-authored-by: Christian Gunderman <gundermanc@google.com> Co-authored-by: Sehoon Shon <sshon@google.com> Co-authored-by: Abhi <43648792+abhipatel12@users.noreply.github.com> Co-authored-by: MD. MOHIBUR RAHMAN <35300157+mrpmohiburrahman@users.noreply.github.com> Co-authored-by: David Pierce <davidapierce@google.com> Co-authored-by: chernistry <73943355+chernistry@users.noreply.github.com> Co-authored-by: euxaristia <25621994+euxaristia@users.noreply.github.com> Co-authored-by: Michael Bleigh <mbleigh@mbleigh.com> Co-authored-by: Adam Weidman <adamfweidman@gmail.com> Co-authored-by: Adam Weidman <adamfweidman@google.com> Co-authored-by: Yuna Seol <yunaseol@gmail.com> Co-authored-by: June <kimjune01@gmail.com> Co-authored-by: Aishanee Shah <aishaneeshah@google.com> Co-authored-by: Jason Matthew Suhari <jasonmatthewsuhari@gmail.com> Co-authored-by: Christopher Thomas <cobekgn@gmail.com> Co-authored-by: Jenna Inouye <jinouye@google.com> Co-authored-by: cynthialong0-0 <82900738+cynthialong0-0@users.noreply.github.com> Co-authored-by: M Junaid Shaukat <154750865+junaiddshaukat@users.noreply.github.com> Co-authored-by: Abhijit Balaji <abhijitbalaji@google.com> Co-authored-by: Mark Griffith <anthraxmilkshake@hotmail.com> Co-authored-by: Jack Wotherspoon <jackwoth@google.com> Co-authored-by: Jesse Rosenstock <jesse.rosenstock@gmail.com> Co-authored-by: Adib234 <30782825+Adib234@users.noreply.github.com> Co-authored-by: Dev Randalpura <devrandalpura@google.com> Co-authored-by: Anjaligarhwal <anjaligarhwal1610@gmail.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: Tanmay Vartak <9002434+TanmayVartak@users.noreply.github.com> Co-authored-by: Jerop Kipruto <jerop@google.com> Co-authored-by: Kevin Zhao <kevin8093@126.com> Co-authored-by: joshualitt <joshualitt@google.com> Co-authored-by: Clay <claygeo6@gmail.com> Co-authored-by: Adam Weidman <65992621+adamfweidman@users.noreply.github.com> Co-authored-by: Mahima Shanware <mahima.shanware@gmail.com> Co-authored-by: codex-bot <spigelly+gh-bot@gmail.com>

feat(evals): extract component-level eval infrastructure

e68234a

gundermanc force-pushed the component-eval branch from 875f54a to e68234a Compare April 8, 2026 17:44

gundermanc added 5 commits April 8, 2026 11:04

Fix errors.

1582663

Add suite and test type.

ca607a5

Fix compile errors.

6d353a6

Revert.

cba2298

Fix yaml error.

30d70e8

gundermanc marked this pull request as ready for review April 8, 2026 22:08

gundermanc requested review from a team as code owners April 8, 2026 22:08

gundermanc had a problem deploying to eval-gate April 8, 2026 22:09 — with GitHub Actions Failure

gundermanc changed the title ~~feat(evals): extract component-level eval infrastructure~~ Generalize evals infra to support more types of evals, organization and queuing of named suites Apr 8, 2026

gemini-code-assist Bot reviewed Apr 8, 2026

View reviewed changes

Comment thread evals/component-test-helper.ts

Comment thread evals/vitest.config.ts

gemini-cli Bot added the status/need-issue Pull requests that need to have an associated issue. label Apr 8, 2026

alisa-alisa reviewed Apr 8, 2026

View reviewed changes

alisa-alisa approved these changes Apr 8, 2026

View reviewed changes

Merge remote-tracking branch 'origin/main' into component-eval

40e6acf

gundermanc temporarily deployed to eval-gate April 8, 2026 23:06 — with GitHub Actions Inactive

scidomino approved these changes Apr 8, 2026

View reviewed changes

gundermanc added this pull request to the merge queue Apr 8, 2026

Merged via the queue into main with commit f1bb2af Apr 9, 2026
29 checks passed

gundermanc deleted the component-eval branch April 9, 2026 00:29

student-ankitpandit pushed a commit to student-ankitpandit/gemini-cli that referenced this pull request Apr 9, 2026

Generalize evals infra to support more types of evals, organization a…

41b5a60

…nd queuing of named suites (google-gemini#24941)

flexponsive pushed a commit to flexponsive/gemini-cli that referenced this pull request Apr 9, 2026

Generalize evals infra to support more types of evals, organization a…

efd84bd

…nd queuing of named suites (google-gemini#24941)

warrenzhu25 pushed a commit to warrenzhu25/gemini-cli that referenced this pull request Apr 9, 2026

Generalize evals infra to support more types of evals, organization a…

9cfb054

…nd queuing of named suites (google-gemini#24941)

kimjune01 added a commit to kimjune01/gemini-cli-claude that referenced this pull request Apr 18, 2026

forge-v2 refactor of PR google-gemini#24941

5d6f954

Blind-blind winner: opus Complexity gate: delta=0 Reviewer: approved

kimjune01 mentioned this pull request Apr 18, 2026

forge-v2: Generalize evals infra to support more types of evals, organization and queuing of named suites kimjune01/gemini-cli-claude#4

Open

github-actions Bot mentioned this pull request Apr 23, 2026

Tool release: Gemini CLI v0.38.2 (was v0.36.0) agent-sh/agnix#750

Closed

Conversation

gundermanc commented Apr 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Apr 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist Bot commented Apr 8, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

alisa-alisa Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

gundermanc Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

alisa-alisa Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

gundermanc Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

alisa-alisa left a comment

Choose a reason for hiding this comment

Uh oh!

alisa-alisa Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

scidomino Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

gundermanc Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Apr 8, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

gundermanc commented Apr 8, 2026 •

edited

Loading

github-actions Bot commented Apr 8, 2026 •

edited

Loading