feat(evaluator): add kmsKeyArn support for custom evaluator by aws-aditya21 · Pull Request #994 · aws/agentcore-cli

aws-aditya21 · 2026-04-27T23:24:21Z

Description

Add optional kmsKeyArn support for custom evaluators, enabling customers to specify a KMS key for evaluator encryption at rest.

Changes (9 files)

src/schema/schemas/agentcore-project.ts — added kmsKeyArn to EvaluatorSchema
src/cli/primitives/EvaluatorPrimitive.ts — added --kms-key-arn CLI flag, updated options, action handler, and createEvaluator
src/cli/tui/screens/evaluator/types.ts — added kms-key-arn step and kmsKeyArn to AddEvaluatorConfig
src/cli/tui/screens/evaluator/useAddEvaluatorWizard.ts — added KMS state, callback, wired into all wizard flows
src/cli/tui/screens/evaluator/AddEvaluatorScreen.tsx — added KMS key ARN text input and confirm field
src/cli/tui/hooks/useCreateEvaluator.ts — passes kmsKeyArn through to primitive
src/cli/commands/import/import-evaluator.ts — toEvaluatorSpec forwards kmsKeyArn from API response
src/cli/aws/agentcore-control.ts — added kmsKeyArn to GetEvaluatorResult
src/cli/commands/import/__tests__/import-evaluator.test.ts — added tests for kmsKeyArn forwarding

Companion PR

CDK constructs: https://github.com/aws/agentcore-l3-cdk-constructs/pull/184

CFN support for EncryptionKeyArn on AWS::BedrockAgentCore::Evaluator is assumed ready per service team confirmation.

Closes #

Type of Change

Testing

How have you tested the change?

I ran npm run test:unit and npm run test:integ
I ran npm run typecheck
I ran npm run lint
If I modified src/assets/, I ran npm run test:update-snapshots and committed the updated snapshots

Checklist

I have read the CONTRIBUTING document
I have added any necessary tests that prove my fix is effective or my feature works
I have updated the documentation accordingly
I have added an appropriate example to the documentation to outline the feature, or no new docs are needed
My changes generate no new warnings
Any dependent changes have been merged and published

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the
terms of your choice.

github-actions · 2026-04-27T23:27:04Z

Coverage Report

Status	Category	Percentage	Covered / Total
🔵	Lines	43.32%	9085 / 20970
🔵	Statements	42.59%	9648 / 22649
🔵	Functions	40.09%	1569 / 3913
🔵	Branches	40.11%	5865 / 14619

Generated in workflow #2839 for commit c7302c0 by the Vitest Coverage Report Action

agentcore-cli-automation · 2026-04-28T00:13:09Z

Field naming inconsistency: kmsKeyArn vs encryptionKeyArn

Other primitives in both agentcore-cli and agentcore-l3-cdk-constructs schemas use encryptionKeyArn for KMS key fields — see MemorySchema (src/schema/schemas/agentcore-project.ts:142) and PolicyEngineSchema (src/schema/schemas/primitives/policy.ts:71). This PR introduces kmsKeyArn only for the Evaluator, which is inconsistent and will be confusing for users editing agentcore.json.

Additionally, CFN uses EncryptionKeyArn on AWS::BedrockAgentCore::Evaluator (per this PR's description), which maps to encryptionKeyArn in camelCase. The companion CDK PR (aws/agentcore-l3-cdk-constructs#184) will almost certainly use encryptionKeyArn to stay consistent with the other primitives — if so, the kmsKeyArn field in the CLI's agentcore.json won't flow through to the CDK construct.

Options:

Rename the project-schema field to encryptionKeyArn to match existing primitives (recommended). The CLI flag --kms-key-arn can stay as-is or be renamed to --encryption-key-arn for consistency.
If there's a deliberate reason to diverge (e.g., the service team named the API field kmsKeyArn), call it out and consider renaming the Memory/Policy fields instead — but that would be a breaking change.

Please confirm the field name with the companion CDK PR before merging, since whatever is in agentcore.json must match what the CDK schema expects.

agentcore-cli-automation · 2026-04-28T00:13:13Z

Unsafe cast in getEvaluator likely masks a wrong SDK field name

src/cli/aws/agentcore-control.ts:545:

kmsKeyArn: (response as unknown as Record<string, unknown>).kmsKeyArn as string | undefined,

This double-cast bypasses TypeScript's checking of the SDK response type. Compare with how getMemoryDetail reads the analogous field directly as memory.encryptionKeyArn (line 415) — typed and safe.

Given that CFN exposes this as EncryptionKeyArn (per the PR description) and the service's SDK convention matches CFN PascalCase → camelCase (e.g. memory.encryptionKeyArn), the SDK field on GetEvaluatorResponse is almost certainly encryptionKeyArn, not kmsKeyArn. If that's correct, this line will silently always return undefined at runtime, so agentcore import evaluator will never preserve the KMS key on imported evaluators — and the existing unit tests won't catch it because they construct GetEvaluatorResult objects directly instead of exercising the SDK → result mapping.

Options:

Use the real SDK field name directly (e.g. kmsKeyArn: response.encryptionKeyArn) once the SDK typings are updated. Drop the as unknown cast so TypeScript verifies the field exists.
If the SDK types aren't updated yet, at minimum add a comment explaining why the cast is needed and add an integration/mapping test that stubs the SDK response and verifies the field is forwarded — so this regression is caught when the SDK types land.

Either way, please verify by inspecting the actual GetEvaluatorResponse shape (or the service's API model) that kmsKeyArn is the correct field name before merging.

agentcore-cli-automation · 2026-04-28T00:13:14Z

KMS key ARN validation regex is too restrictive

The regex used in both EvaluatorPrimitive.ts (non-interactive flag validation) and AddEvaluatorScreen.tsx (TUI input validation):

/^arn:aws(?:|-cn|-us-gov):kms:[a-zA-Z0-9-]*:[0-9]{12}:key\/[a-zA-Z0-9-]{36}$/

will reject several valid KMS key ARN formats:

Alias ARNs — arn:aws:kms:us-east-1:123456789012:alias/my-key. KMS alias ARNs are commonly used and accepted by most AWS services for encryption configuration.
Other AWS partitions — aws-iso, aws-iso-b, aws-iso-e, aws-iso-f are not matched. Less critical but worth noting.
Custom key store keys — may have IDs that don't fit the 36-char UUID pattern.

Additionally, duplicating the same regex in two places risks them drifting; consider extracting to a shared helper (e.g. isValidKmsKeyArn) alongside the other schema helpers.

Options:

Loosen the regex to accept both key/... and alias/... resource parts, e.g. ^arn:aws[a-z0-9-]*:kms:[a-z0-9-]+:[0-9]{12}:(key|alias)\/.+$, and extract it to a shared helper.
Skip client-side ARN format validation entirely (let the AWS control-plane API reject bad ARNs on deploy). Other primitives in this repo don't appear to do client-side KMS ARN regex validation.
Keep the strict validation but at least extract the regex to a single constant so both call sites stay in sync.

aws-aditya21 · 2026-04-28T00:17:06Z

KMS key ARN validation regex is too restrictive

The regex used in both EvaluatorPrimitive.ts (non-interactive flag validation) and AddEvaluatorScreen.tsx (TUI input validation):
/^arn:aws(?:|-cn|-us-gov):kms:[a-zA-Z0-9-]*:[0-9]{12}:key\/[a-zA-Z0-9-]{36}$/
will reject several valid KMS key ARN formats:

Alias ARNs — arn:aws:kms:us-east-1:123456789012:alias/my-key. KMS alias ARNs are commonly used and accepted by most AWS services for encryption configuration.

Other AWS partitions — aws-iso, aws-iso-b, aws-iso-e, aws-iso-f are not matched. Less critical but worth noting.

Custom key store keys — may have IDs that don't fit the 36-char UUID pattern.

Additionally, duplicating the same regex in two places risks them drifting; consider extracting to a shared helper (e.g. isValidKmsKeyArn) alongside the other schema helpers.

Options:

Loosen the regex to accept both key/... and alias/... resource parts, e.g. ^arn:aws[a-z0-9-]*:kms:[a-z0-9-]+:[0-9]{12}:(key|alias)\/.+$, and extract it to a shared helper.

Skip client-side ARN format validation entirely (let the AWS control-plane API reject bad ARNs on deploy). Other primitives in this repo don't appear to do client-side KMS ARN regex validation.

Keep the strict validation but at least extract the regex to a single constant so both call sites stay in sync.

agentcore-cli-automation · 2026-04-28T00:20:48Z

Reviewed the PR. The three existing automation comments (field naming kmsKeyArn vs encryptionKeyArn, unsafe SDK cast in getEvaluator, and overly-restrictive KMS ARN regex) cover the serious issues I found — no additional blockers from me. The naming issue in particular should be resolved against the companion CDK PR before merging, since the existing MemorySchema in agentcore-l3-cdk-constructs uses encryptionKeyArn and a mismatch would cause the field to be silently stripped at deploy time.

github-actions · 2026-05-12T16:51:42Z

Package Tarball

aws-agentcore-0.13.1.tgz

How to install

npm install https://github.com/aws/agentcore-cli/releases/download/pr-994-tarball/aws-agentcore-0.13.1.tgz

The lock file was out of sync after dependency bumps on main were merged, causing npm ci to fail in CI.

Reverts @opentelemetry/exporter-metrics-otlp-http ^0.217.0 back to ^0.214.0 and secretlint ^13.0.0 back to ^12.2.0 — these were accidentally included in the feature commit from unmerged dependabot PRs and introduce high-severity protobufjs vulnerabilities. Restores fast-xml-parser and @aws-sdk/xml-builder overrides that were also inadvertently removed. Fixes Prettier formatting on agentcore-project.ts import lines.

tejaskash

LGTM

Cherry-picks non-breaking changes from main: - docs: add telemetry instrumentation guide (#1197) - chore: replace PAT tokens with GitHub App token (#1198) - fix: add batch eval, recommendation, CloudWatch Logs permissions (#1113) - feat(evaluator): add kmsKeyArn support (#994) - chore: replace github.token with GitHub App token (#1210) - fix: sync-preview workflow rewrite (#1078) Skipped (requires dedicated effort due to Result type refactor): - refactor: unify result types with Result<T, E> (#1125) - feat: instrument telemetry for create/deploy (#1202, #1206) - feat: record command attrs on failure (#1204)

* docs: add telemetry instrumentation guide (#1197) * chore: replace PAT tokens with GitHub App token (#1198) Replace secrets.PAT_TOKEN and secrets.AUTOMATION_ACCOUNT_PAT_TOKEN with short-lived tokens generated by the agentcore-devx-automation GitHub App (ID: 3637953) via actions/create-github-app-token@v1. This improves security by using ephemeral tokens scoped to the installation rather than long-lived personal access tokens. Requires adding repo variable APP_ID=3637953 and repo secret APP_PRIVATE_KEY with the app's RSA private key. * fix: add batch eval, recommendation, and CloudWatch Logs write permissions to docs (#1113) * feat: instrument telemetry for create command (CLI + TUI) (#1202) Add telemetry recording to the create command in both CLI and TUI paths: - CLI: wrap handleCreateCLI with runCliCommand to emit CreateAttrs on success/failure - TUI: wrap useCreateFlow's run() with withCommandRunTelemetry - Add telemetry assertions to existing integration tests (frameworks + edge cases) * chore: replace all github.token/GITHUB_TOKEN with GitHub App token Replaces all occurrences of \${{ github.token }} and \${{ secrets.GITHUB_TOKEN }} across .github/workflows/ with a per-job GitHub App token generated via actions/create-github-app-token@v1 using vars.APP_ID and secrets.APP_PRIVATE_KEY. * fix: bump versions to resolve security audit failure * revert: keep pr-title.yml using GITHUB_TOKEN (read-only access sufficient) * feat(evaluator): add kmsKeyArn support for custom evaluator (#994) * feat(evaluator): Add kmsKeyArn support for custom evaluator * fix: sync package-lock.json with package.json The lock file was out of sync after dependency bumps on main were merged, causing npm ci to fail in CI. * fix: revert unrelated dep bumps and fix formatting Reverts @opentelemetry/exporter-metrics-otlp-http ^0.217.0 back to ^0.214.0 and secretlint ^13.0.0 back to ^12.2.0 — these were accidentally included in the feature commit from unmerged dependabot PRs and introduce high-severity protobufjs vulnerabilities. Restores fast-xml-parser and @aws-sdk/xml-builder overrides that were also inadvertently removed. Fixes Prettier formatting on agentcore-project.ts import lines. * fix: sync package-lock.json with updated dependencies --------- Co-authored-by: notgitika <gitijh@gmail.com> * refactor: unify result types with discriminated Result<T, E> union (#1125) * refactor: unify result types with discriminated Result<T, E> union Introduce a shared Result<T, E> type (inspired by Rust's Result) that replaces ad-hoc { success: boolean; error?: string } patterns across the codebase. Key changes: - Add src/lib/types.ts with Result<T, E> discriminated union type - Add toError() helper in src/cli/errors.ts for catch blocks - Migrate all command, operation, and primitive result types to Result<T> - Error field is now Error (not string) on the failure branch - Data fields only exist on the success branch (proper narrowing) - Update all consumers to narrow before accessing branch-specific fields - Update test assertions to match new Error objects and add narrowing * docs: update AGENTS.md and telemetry README to reflect Result<T, E> type * feat: record command attrs on telemetry failure via fallbackAttrs (#1204) * feat: record command attrs on telemetry failure via fallbackAttrs Add optional fallbackAttrs parameter to client.withCommandRun so command-specific attributes are recorded even when the callback throws. - client.ts: accept fallbackAttrs, use on failure instead of {} - client.ts: run resilientParse on all non-empty attrs (not just success) - cli-command-run.ts: withCommandRunTelemetry passes attrs as fallbackAttrs - cli-command-run.ts: runCliCommand accepts optional knownAttrs param - command.tsx: extract knownAttrs upfront, pass to runCliCommand - client.test.ts: add unit tests for fallbackAttrs behavior - create-edge-cases.test.ts: assert attrs present on failure entry * chore: rebase onto mainline * fix: sync-preview pushes directly on clean merge, PRs only on conflict (#1078) * fix: sync-preview workflow restores version instead of ignoring files Instead of keeping preview's entire package.json/package-lock.json (which discards new deps, scripts, etc. from main), accept main's content and surgically restore only the version field to preview's value after merge. * fix: push directly to preview on clean merge via GitHub App bypass Use agentcore-devx-automation app token to bypass branch protection and push directly when the merge is clean (or only version conflicts). Only creates a PR when there are real conflicts in other files. * chore: use app-slug instead of app-id for token generation * fix: address review feedback on sync-preview workflow - Pass PREVIEW_VERSION via env var instead of string interpolation in node -e scripts (safer against special chars) - Make git add of package-lock.json conditional on file existence to match the earlier -f guard - Replace loose title search for dedup with headRefName prefix filter to avoid false positives from unrelated PRs - Clarify why package.json/package-lock.json are special-cased (preview carries a different version string that needs preserving) * fix: restore preview-owned files after sync merge Adds a step to restore schemas/agentcore.schema.v1.json and CHANGELOG.md to preview's versions after merging main. These files are auto-generated during preview releases — schema-check CI rejects direct modifications to schemas/, and CHANGELOG.md tracks preview releases separately. * fix: use app-id instead of app-slug for GitHub App token Aligns with the pattern in PR #1210 and ci-failure-issue.yml. * feat: instrument telemetry for deploy command (CLI + TUI) (#1206) * chore: sync safe commits from main into preview Cherry-picks non-breaking changes from main: - docs: add telemetry instrumentation guide (#1197) - chore: replace PAT tokens with GitHub App token (#1198) - fix: add batch eval, recommendation, CloudWatch Logs permissions (#1113) - feat(evaluator): add kmsKeyArn support (#994) - chore: replace github.token with GitHub App token (#1210) - fix: sync-preview workflow rewrite (#1078) Skipped (requires dedicated effort due to Result type refactor): - refactor: unify result types with Result<T, E> (#1125) - feat: instrument telemetry for create/deploy (#1202, #1206) - feat: record command attrs on failure (#1204) * chore: merge main into preview with Result refactor Brings in all remaining main commits including: - refactor: unify result types with Result<T, E> (#1125) - feat: instrument telemetry for deploy command (#1206) - feat: record command attrs on failure (#1204) - feat: instrument telemetry for create command (#1202) Adapts preview's deploy flow to use OperationResult with string errors for compatibility with the telemetry layer. --------- Co-authored-by: Hweinstock <42325418+Hweinstock@users.noreply.github.com> Co-authored-by: Aidan Daly <99039782+aidandaly24@users.noreply.github.com> Co-authored-by: Gitika <53349492+notgitika@users.noreply.github.com> Co-authored-by: Aidan Daly <aidandal@amazon.com> Co-authored-by: Harrison Weinstock <hkobew@amazon.com> Co-authored-by: aws-aditya21 <saivenki@amazon.com> Co-authored-by: notgitika <gitijh@gmail.com>

github-actions Bot added size/s PR size: S agentcore-harness-reviewing AgentCore Harness review in progress labels Apr 27, 2026

aws-aditya21 changed the title ~~feat(evaluator): Add kmsKeyArn support for custom evaluator~~ feat(evaluator): add kmsKeyArn support for custom evaluator Apr 27, 2026

github-actions Bot removed the agentcore-harness-reviewing AgentCore Harness review in progress label Apr 27, 2026

aws-aditya21 force-pushed the feat/evaluator-kms-key-arn branch from ba4c55c to fe5cd4b Compare April 27, 2026 23:46

github-actions Bot added size/m PR size: M and removed size/s PR size: S labels Apr 27, 2026

aws deleted a comment from agentcore-cli-automation Apr 28, 2026

aws-aditya21 closed this Apr 28, 2026

aws-aditya21 reopened this Apr 28, 2026

github-actions Bot added size/m PR size: M agentcore-harness-reviewing AgentCore Harness review in progress and removed size/m PR size: M labels Apr 28, 2026

aws deleted a comment from agentcore-cli-automation Apr 28, 2026

github-actions Bot removed the agentcore-harness-reviewing AgentCore Harness review in progress label Apr 28, 2026

aws-aditya21 closed this Apr 28, 2026

aws-aditya21 reopened this Apr 28, 2026

github-actions Bot added size/m PR size: M agentcore-harness-reviewing AgentCore Harness review in progress and removed size/m PR size: M labels Apr 28, 2026

github-actions Bot removed the agentcore-harness-reviewing AgentCore Harness review in progress label Apr 28, 2026

aws-aditya21 force-pushed the feat/evaluator-kms-key-arn branch from fe5cd4b to 9bc2d70 Compare April 29, 2026 23:47

github-actions Bot added size/m PR size: M and removed size/m PR size: M labels Apr 29, 2026

notgitika temporarily deployed to e2e-testing May 12, 2026 16:44 — with GitHub Actions Inactive

github-actions Bot added size/m PR size: M and removed size/m PR size: M labels May 12, 2026

notgitika temporarily deployed to e2e-testing May 12, 2026 17:04 — with GitHub Actions Inactive

padmak30 approved these changes May 12, 2026

View reviewed changes

aws-aditya21 and others added 3 commits May 12, 2026 14:11

feat(evaluator): Add kmsKeyArn support for custom evaluator

7b599b0

fix: sync package-lock.json with package.json

6b3c21d

The lock file was out of sync after dependency bumps on main were merged, causing npm ci to fail in CI.

padmak30 previously approved these changes May 12, 2026

View reviewed changes

notgitika dismissed padmak30’s stale review via 64421a9 May 12, 2026 18:15

notgitika force-pushed the feat/evaluator-kms-key-arn branch from ed5651a to 64421a9 Compare May 12, 2026 18:15

padmak30 previously approved these changes May 12, 2026

View reviewed changes

github-actions Bot added size/m PR size: M and removed size/m PR size: M labels May 12, 2026

notgitika had a problem deploying to e2e-testing May 12, 2026 18:31 — with GitHub Actions Failure

fix: sync package-lock.json with updated dependencies

c7302c0

notgitika dismissed padmak30’s stale review via c7302c0 May 12, 2026 18:33

tejaskash approved these changes May 12, 2026

View reviewed changes

github-actions Bot added size/m PR size: M and removed size/m PR size: M labels May 12, 2026

notgitika temporarily deployed to e2e-testing May 12, 2026 19:19 — with GitHub Actions Inactive

notgitika merged commit 7d27f47 into main May 12, 2026
23 of 24 checks passed

notgitika deleted the feat/evaluator-kms-key-arn branch May 12, 2026 19:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(evaluator): add kmsKeyArn support for custom evaluator#994

feat(evaluator): add kmsKeyArn support for custom evaluator#994
notgitika merged 4 commits into
mainfrom
feat/evaluator-kms-key-arn

aws-aditya21 commented Apr 27, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Apr 27, 2026 •

edited

Loading

Uh oh!

agentcore-cli-automation commented Apr 28, 2026

Uh oh!

agentcore-cli-automation commented Apr 28, 2026

Uh oh!

agentcore-cli-automation commented Apr 28, 2026

Uh oh!

aws-aditya21 commented Apr 28, 2026

Uh oh!

agentcore-cli-automation commented Apr 28, 2026

Uh oh!

github-actions Bot commented May 12, 2026

Uh oh!

tejaskash left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

aws-aditya21 commented Apr 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Changes (9 files)

Companion PR

Type of Change

Testing

Checklist

Uh oh!

github-actions Bot commented Apr 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Coverage Report

Uh oh!

agentcore-cli-automation commented Apr 28, 2026

Uh oh!

agentcore-cli-automation commented Apr 28, 2026

Uh oh!

agentcore-cli-automation commented Apr 28, 2026

Uh oh!

aws-aditya21 commented Apr 28, 2026

Uh oh!

agentcore-cli-automation commented Apr 28, 2026

Uh oh!

github-actions Bot commented May 12, 2026

Package Tarball

How to install

Uh oh!

tejaskash left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

aws-aditya21 commented Apr 27, 2026 •

edited

Loading

github-actions Bot commented Apr 27, 2026 •

edited

Loading