fix(evals): update eval tests for invoke_agent telemetry and project-scoped memory

[SandyTao520]

Summary

Fix 8 behavioral eval tests that fail 100% across all models and all nightly shards due to two recent infrastructure changes: invoke_agent telemetry naming and project-scoped save_memory.

Details

Problem

Two root causes were causing consistent eval failures in nightly runs:

1. invoke_agent telemetry change: Subagent calls (e.g., generalist, cli_help, save_memory agent) now appear in telemetry as function_name: 'invoke_agent' instead of the agent's name. Evals using waitForToolCall('agent_name') or readToolLogs().find(name === 'agent_name') could never match.

2. Project-scoped save_memory: The save_memory tool now supports scope: 'project' for workspace-specific facts. Three eval tests that asserted save_memory should NOT be called for workspace info (db schema, build artifacts, entry point) were wrong — the model correctly saves these with scope: 'project'.

Fix

File	Tests Fixed	Change
save_memory.eval.ts	3 workspace tests	Assert save_memory IS called with scope='project'
save_memory.eval.ts	2 memory manager tests	Check invoke_agent with save_memory in args
generalist_agent.eval.ts	1 test	Check invoke_agent with agent_name='generalist' in args
cli_help_delegation.eval.ts	1 test	Check invoke_agent with agent_name='cli_help' in args

The subagents.eval.ts file was already correctly using AGENT_TOOL_NAME (invoke_agent) — no changes needed there.

Related Issues

Fixes #18007

How to Validate

1. Check the next nightly eval run — these 8 tests should no longer be 100% failures.
2. Alternatively, run locally with a specific model:

   GEMINI_MODEL=gemini-2.5-flash npm run test:always_passing_evals -- --testPathPattern save_memory

Pre-Merge Checklist

• Updated relevant documentation and README (if needed)
• Noted breaking changes (if any)
• Validated on required platforms/methods:
  • MacOS
    • npm run

[github-actions]

🛑 Action Required: Evaluation Approval

Steering changes have been detected in this PR. To prevent regressions, a maintainer must approve the evaluation run before this PR can be merged.

Maintainers:

1. Go to the Workflow Run Summary.
2. Click the yellow 'Review deployments' button.
3. Select the 'eval-gate' environment and click 'Approve'.

Once approved, the evaluation results will be posted here automatically.

[github-actions]

Size Change: -4 B (0%)

Total Size: 33.6 MB

Filename	Size	Change
./bundle/chunk-DF5F567H.js	0 B	-3.43 MB (removed)	🏆
./bundle/chunk-LTOHRV5G.js	0 B	-14.5 MB (removed)	🏆
./bundle/chunk-TPBGQAPH.js	0 B	-3.8 kB (removed)	🏆
./bundle/core-SYHUAHZM.js	0 B	-46.6 kB (removed)	🏆
./bundle/devtoolsService-7HOCY4ME.js	0 B	-28.4 kB (removed)	🏆
./bundle/gemini-3QDRJNOT.js	0 B	-553 kB (removed)	🏆
./bundle/interactiveCli-K5OSRQ2V.js	0 B	-1.29 MB (removed)	🏆
./bundle/oauth2-provider-ZTNQ6M34.js	0 B	-9.16 kB (removed)	🏆
./bundle/chunk-622BT5DZ.js	3.8 kB	+3.8 kB (new file)	🆕
./bundle/chunk-AXTCGYJD.js	3.43 MB	+3.43 MB (new file)	🆕
./bundle/chunk-YIVEUSUM.js	14.5 MB	+14.5 MB (new file)	🆕
./bundle/core-5EGOUUCC.js	46.6 kB	+46.6 kB (new file)	🆕
./bundle/devtoolsService-UOI3NM5M.js	28.4 kB	+28.4 kB (new file)	🆕
./bundle/gemini-2L3I7FRP.js	553 kB	+553 kB (new file)	🆕
./bundle/interactiveCli-Q7IO2JD5.js	1.29 MB	+1.29 MB (new file)	🆕
./bundle/oauth2-provider-GB2JJEKA.js	9.16 kB	+9.16 kB (new file)	🆕

ℹ️ View Unchanged

Filename	Size	Change
./bundle/bundled/third_party/index.js	8 MB	0 B
./bundle/chunk-34MYV7JD.js	2.45 kB	0 B
./bundle/chunk-5AUYMPVF.js	858 B	0 B
./bundle/chunk-5PS3AYFU.js	1.18 kB	0 B
./bundle/chunk-664ZODQF.js	124 kB	0 B
./bundle/chunk-DAHVX5MI.js	206 kB	0 B
./bundle/chunk-IUUIT4SU.js	56.5 kB	0 B
./bundle/chunk-PC3Y4R7E.js	1.97 MB	0 B
./bundle/chunk-RJTRUG2J.js	39.8 kB	0 B
./bundle/cleanup-64C4QZRA.js	0 B	-932 B (removed)	🏆
./bundle/devtools-36NN55EP.js	696 kB	0 B
./bundle/dist-T73EYRDX.js	356 B	0 B
./bundle/events-XB7DADIJ.js	418 B	0 B
./bundle/gemini.js	4.97 kB	0 B
./bundle/getMachineId-bsd-TXG52NKR.js	1.55 kB	0 B
./bundle/getMachineId-darwin-7OE4DDZ6.js	1.55 kB	0 B
./bundle/getMachineId-linux-SHIFKOOX.js	1.34 kB	0 B
./bundle/getMachineId-unsupported-5U5DOEYY.js	1.06 kB	0 B
./bundle/getMachineId-win-6KLLGOI4.js	1.72 kB	0 B
./bundle/memoryDiscovery-R4AKKXHF.js	980 B	0 B
./bundle/multipart-parser-KPBZEGQU.js	11.7 kB	0 B
./bundle/node_modules/@google/gemini-cli-devtools/dist/client/main.js	222 kB	0 B
./bundle/node_modules/@google/gemini-cli-devtools/dist/src/_client-assets.js	229 kB	0 B
./bundle/node_modules/@google/gemini-cli-devtools/dist/src/index.js	13.4 kB	0 B
./bundle/node_modules/@google/gemini-cli-devtools/dist/src/types.js	132 B	0 B
./bundle/sandbox-macos-permissive-open.sb	890 B	0 B
./bundle/sandbox-macos-permissive-proxied.sb	1.31 kB	0 B
./bundle/sandbox-macos-restrictive-open.sb	3.36 kB	0 B
./bundle/sandbox-macos-restrictive-proxied.sb	3.56 kB	0 B
./bundle/sandbox-macos-strict-open.sb	4.82 kB	0 B
./bundle/sandbox-macos-strict-proxied.sb	5.02 kB	0 B
./bundle/src-QVCVGIUX.js	47 kB	0 B
./bundle/tree-sitter-7U6MW5PS.js	274 kB	0 B
./bundle/tree-sitter-bash-34ZGLXVX.js	1.84 MB	0 B
./bundle/cleanup-WX66ORVJ.js	932 B	+932 B (new file)	🆕

_{compressed-size-action}

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses consistent failures in behavioral evaluation tests caused by recent infrastructure changes. By updating how subagent invocations and memory scoping are verified, the changes ensure that the test suite accurately reflects the current system behavior regarding telemetry and project-scoped memory management.

Highlights

• Telemetry Adaptation: Updated behavioral evaluation tests to account for the 'invoke_agent' telemetry naming convention, ensuring subagent calls are correctly identified by inspecting tool arguments.
• Project-Scoped Memory: Adjusted 'save_memory' assertions to validate that workspace-specific information is correctly saved with a 'project' scope, reflecting recent infrastructure updates.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request updates evaluation tests in cli_help_delegation.eval.ts, generalist_agent.eval.ts, and save_memory.eval.ts to support the invoke_agent tool and ensure memory is saved with a 'project' scope. Feedback was provided to replace regex-based argument validation with JSON parsing in save_memory.eval.ts to ensure consistency with other tests and prevent potential false positives.

[gemini-code-assist]

The current predicate uses a loose regex check on the entire stringified arguments of the invoke_agent tool. This can lead to false positives if the agent name is different but the request string happens to contain 'save_memory'. As per repository rules, toolRequest.args is a JSON string and must be parsed using JSON.parse() before its properties can be accessed. It is safer to parse the JSON and explicitly check the agent_name property.

        (args) => {
          try {
            const parsed = JSON.parse(args);
            return parsed.agent_name === 'save_memory' && /vitest/i.test(args);
          } catch {
            return false;
          }
        },

References

1. The toolRequest.args property is a JSON string, not an object. It must be parsed using JSON.parse() before its properties can be accessed.

[gemini-code-assist]

Similar to the previous test, using a simple regex on the raw arguments string for invoke_agent is imprecise. Following the repository rule that toolRequest.args is a JSON string, it should be explicitly parsed to verify that the agent_name is save_memory to avoid false positives from other subagent calls.

        (args) => {
          try {
            const parsed = JSON.parse(args);
            return parsed.agent_name === 'save_memory';
          } catch {
            return false;
          }
        },

References

1. The toolRequest.args property is a JSON string, not an object. It must be parsed using JSON.parse() before its properties can be accessed.