diff --git a/.ai/active/SPRINT_PACKET.md b/.ai/active/SPRINT_PACKET.md index 4c6e5b9..74fe635 100644 --- a/.ai/active/SPRINT_PACKET.md +++ b/.ai/active/SPRINT_PACKET.md @@ -2,134 +2,115 @@ ## Sprint Title -Sprint 5T: External Secret-Manager Integration For Gmail Credentials +Sprint 5U: Project Truth Synchronization After Gmail Secret Externalization ## Sprint Type -feature +refactor ## Sprint Reason -Sprint 5S confirmed the repo is on track and synchronized the truth artifacts through Sprint 5R. The next narrow risk is no longer Gmail auth correctness inside the app database; it is secret-storage boundary quality. The roadmap and current-state docs both identify external secret-manager integration as the strongest next Gmail auth-adjacent seam before broader Gmail scope, Calendar, or UI work. +Sprint 5T is implemented and the repo remains on track, but the live truth artifacts have drifted again. `ARCHITECTURE.md` now reflects Sprint 5T, while `ROADMAP.md` and `.ai/handoff/CURRENT_STATE.md` are still anchored around Sprint 5R. Before opening the next Gmail follow-up seam such as `legacy_db_v0` removal, Control Tower needs the planning and handoff artifacts reset to the accepted repo state. ## Sprint Intent -Extend the existing protected Gmail credential seam so `gmail_account_credentials` can resolve secrets through one explicit external secret-manager boundary, while keeping Gmail account reads secret-free and preserving the existing single-message ingestion contract. +Synchronize the live truth artifacts with the implemented and review-passed repo state through Sprint 5T, so future planning, handoff, and review work all start from accurate architecture, roadmap, and current-state documents. ## Git Instructions -- Branch Name: `codex/sprint-5t-gmail-external-secret-manager` +- Branch Name: `codex/sprint-5u-project-truth-sync-after-gmail-secret-externalization` - Base Branch: `main` - PR Strategy: one sprint branch, one PR, no stacked PRs unless Control Tower explicitly opens a follow-up sprint - Merge Policy: squash merge only after reviewer `PASS` and explicit Control Tower merge approval ## Why This Sprint -- Sprint 5O opened the first narrow read-only Gmail account and single-message ingestion seam. -- Sprint 5P removed plaintext credential storage from the normal `gmail_accounts` surface. -- Sprint 5Q added refresh-token-backed renewal. -- Sprint 5R added rotated refresh-token persistence. -- Sprint 5S synchronized project truth and explicitly identified external secret-manager integration as the next narrow Gmail auth-adjacent seam. -- The next safe step is to externalize the protected credential storage boundary itself without widening into Gmail search, sync, attachments, Calendar, or UI. +- Sprint 5S last synchronized truth through Sprint 5R. +- Sprint 5T changed the accepted connector boundary materially: + - the primary Gmail credential path is now external-secret-backed + - `gmail_account_credentials` now persists locator metadata on the primary path + - a narrow `legacy_db_v0` transition path now exists for older rows +- `ARCHITECTURE.md` already reflects that accepted state, but `ROADMAP.md` and `.ai/handoff/CURRENT_STATE.md` still describe the repo through Sprint 5R and still describe external secret-manager integration as future work. +- Planning from that stale truth would create avoidable scope and review drift for the next Gmail sprint. ## In Scope -- Add schema and migration support only as needed to support an external-secret-backed credential locator, for example: - - secret reference fields on `gmail_account_credentials` - - narrow metadata needed to distinguish local protected credentials from externally stored credentials during the transition -- Define typed contract changes only where needed for: - - Gmail account connect writes if secret-manager-backed credential writes require a narrow new write shape - - deterministic Gmail ingestion failure responses when external secret resolution fails -- Implement a narrow external secret-manager seam that: - - writes Gmail credential material through one explicit secret-manager adapter boundary - - persists only the non-secret locator or reference metadata in the application database - - resolves credentials through that adapter for the existing single-message ingestion and token-renewal path - - supports deterministic rotation-capable credential updates through the same externalized path - - preserves secret-free Gmail account reads and per-user isolation -- Support one explicit runtime configuration path for the secret-manager adapter, with one deterministic local fallback only if needed for tests -- Add unit and integration tests for: - - external secret reference persistence - - absence of secret material in Gmail account responses and normal table reads - - successful single-message Gmail ingestion through the externalized credential path - - successful refresh-token renewal and rotated refresh-token persistence through the externalized path - - deterministic failure when secret resolution or secret update fails - - per-user isolation - - stable response shape +- Audit the accepted implemented slice from the repo and passed sprint reports through Sprint 5T. +- Update `ARCHITECTURE.md` only if needed so it accurately describes the implemented seams through: + - externalized Gmail secret-manager-backed credential storage + - secret-free Gmail account reads + - refresh-token renewal and rotated refresh-token persistence through the externalized seam + - the narrow `legacy_db_v0` transition boundary +- Update `ROADMAP.md` so: + - completed and current milestone state reflects the accepted repo state through Sprint 5T + - the next delivery focus is framed from the actual shipped Gmail externalized-secret baseline + - stale “next delivery focus” language that still treats external secret-manager integration as future work is corrected +- Update `.ai/handoff/CURRENT_STATE.md` so: + - implemented areas and current boundaries reflect the repo through Sprint 5T + - the current milestone position is correct + - the immediate next move matches the next narrow sprint boundary after truth sync +- Update `BUILD_REPORT.md` with the truth-sync evidence and exact files corrected. ## Out of Scope -- No Gmail search. -- No mailbox sync or backfill jobs. -- No attachment ingestion. -- No write-capable Gmail actions. +- No schema changes. +- No API changes. +- No runtime code changes. +- No Gmail search, mailbox sync, attachments, or write actions. - No Calendar connector scope. -- No OAuth UI or callback handling. -- No broader connector-secret abstraction for other providers yet. -- No compile contract changes. +- No `legacy_db_v0` removal yet. - No runner-style orchestration. - No UI work. ## Required Deliverables -- Migration updating the Gmail protected-credential seam to support external secret-manager references. -- Stable Gmail account contracts that remain secret-free on reads. -- One explicit external secret-manager adapter path used by Gmail credential reads and writes. -- Updated single-message Gmail ingestion plus refresh/rotation path running through the externalized credential seam. -- Unit and integration coverage for reference persistence, secret resolution, renewal/rotation continuity, failure handling, and isolation. -- Updated `BUILD_REPORT.md` with exact verification results and explicit deferred scope. +- Updated `ARCHITECTURE.md` aligned to the implemented repo state through Sprint 5T. +- Updated `ROADMAP.md` with correct completed/current/next milestone sequencing. +- Updated `.ai/handoff/CURRENT_STATE.md` reflecting the actual shipped state and immediate next move. +- Updated `BUILD_REPORT.md` describing exactly which truth artifacts were synchronized and what evidence was used. ## Acceptance Criteria -- The repo no longer depends on application-table-stored Gmail secret material for the primary protected credential path. -- `gmail_account_credentials` persists only non-secret reference or locator data for the externalized path. -- Gmail account list and detail responses remain secret-free. -- The existing single-message Gmail ingestion path still works through the external secret-manager seam. -- Refresh-token renewal and rotated refresh-token persistence still work through the externalized credential seam. -- Secret-resolution or secret-update failures fail deterministically and do not corrupt Gmail account, task workspace, or artifact state. -- `./.venv/bin/python -m pytest tests/unit` passes. -- `./.venv/bin/python -m pytest tests/integration` passes. -- No Gmail search, sync, attachments, write actions, Calendar, compile-contract, runner, or UI scope enters the sprint. +- `ARCHITECTURE.md` accurately describes the shipped Gmail seam as external-secret-backed on the primary path, secret-free on reads, renewal-capable, rotation-capable, and still narrow/read-only. +- `ROADMAP.md` no longer claims the next Gmail auth-adjacent seam is external secret-manager integration. +- `.ai/handoff/CURRENT_STATE.md` no longer describes external secret-manager integration as unimplemented. +- Truth artifacts clearly distinguish implemented Gmail externalization behavior from later planned Gmail breadth and from the still-deferred `legacy_db_v0` cleanup sprint. +- No runtime, schema, API, connector-breadth, runner, or UI changes appear in the sprint diff. ## Implementation Constraints -- Keep the auth-storage change narrow and boring. -- Reuse the existing `gmail_accounts` and `gmail_account_credentials` seam rather than introducing a second connector-account model. -- Preserve secret-free Gmail account reads. -- Keep the existing selected-message Gmail ingestion contract stable. -- Use one explicit secret-manager adapter boundary only; do not generalize it to every future provider in this sprint. -- If tests need a local adapter, keep it as a narrow testable implementation detail rather than a second product seam. +- Keep this sprint documentation-only and boring. +- Use accepted repo state and passed sprint reports as evidence, not aspiration. +- Prefer explicit “implemented now” versus “planned later” boundaries. +- If a truth artifact cannot be updated confidently from accepted evidence, narrow the statement rather than guessing. +- Do not widen into product changes just because the roadmap or handoff text is stale. ## Suggested Work Breakdown -1. Add the schema or migration changes required for external secret references. -2. Define any minimal Gmail write-contract updates needed for secret-manager-backed credential writes. -3. Implement one explicit secret-manager adapter for Gmail credential create, read, refresh, and rotation update paths. -4. Keep Gmail account reads secret-free and stable. -5. Add deterministic failure handling for secret resolution and secret update failures. -6. Add unit and integration tests. -7. Update `BUILD_REPORT.md` with executed verification. +1. Audit the implemented repo state and accepted sprint reports through Sprint 5T. +2. Update `ARCHITECTURE.md` only where it still lags the accepted Gmail externalization seam. +3. Update `ROADMAP.md` to reflect actual completed and current milestone state. +4. Update `.ai/handoff/CURRENT_STATE.md` to reflect actual current state and the immediate next move. +5. Update `BUILD_REPORT.md` with exact truth-sync evidence and scope confirmation. ## Build Report Requirements `BUILD_REPORT.md` must include: -- the exact Gmail credential schema and contract changes introduced -- the external secret-manager adapter rule used -- exact commands run -- unit and integration test results -- one example Gmail account response proving secret-free reads remain intact -- one example Gmail ingestion response through the externalized credential path -- what remains intentionally deferred to later milestones +- exactly which truth artifacts were updated +- which accepted reports or repo evidence were used +- the specific stale statements that were corrected +- confirmation that no runtime or schema changes were made +- what remains intentionally deferred after truth synchronization ## Review Focus `REVIEW_REPORT.md` should verify: -- the sprint stayed limited to external secret-manager integration for the Gmail credential seam -- the primary Gmail protected credential path no longer depends on application-table-stored secret material -- Gmail account reads remain secret-free -- single-message Gmail ingestion plus refresh/rotation still work through the externalized seam -- failure handling, isolation, and response stability are test-backed -- no hidden Gmail search, sync, attachments, write actions, Calendar, compile-contract, runner, or UI scope entered the sprint +- the sprint stayed documentation-only +- `ARCHITECTURE.md`, `ROADMAP.md`, and `.ai/handoff/CURRENT_STATE.md` now match the implemented repo state through Sprint 5T +- the shipped Gmail external secret-manager seam and its current narrow boundary are documented accurately +- milestone sequencing is truthful and current +- no hidden runtime, schema, API, connector-breadth, runner, or UI scope entered the sprint ## Exit Condition -This sprint is complete when the repo resolves Gmail secrets through one explicit external secret-manager boundary for the existing read-only single-message ingestion seam, preserves renewal and rotation behavior through that boundary, and verifies the full path with Postgres-backed tests while broader connector behavior remains deferred. +This sprint is complete when the project truth artifacts accurately describe the implemented repo state through Sprint 5T and future planning can proceed from synchronized architecture, roadmap, and current-state documents. diff --git a/.ai/handoff/CURRENT_STATE.md b/.ai/handoff/CURRENT_STATE.md index aa6ef14..ddca142 100644 --- a/.ai/handoff/CURRENT_STATE.md +++ b/.ai/handoff/CURRENT_STATE.md @@ -2,15 +2,15 @@ ## Canonical Truth -- The working repo state is current through Sprint 5R, including narrow PDF/DOCX/RFC822 artifact ingestion, read-only Gmail account persistence and single-message ingestion, protected Gmail credential storage, refresh-token renewal, and rotated refresh-token persistence. +- The working repo state is current through Sprint 5T, including narrow PDF/DOCX/RFC822 artifact ingestion, read-only Gmail account persistence and single-message ingestion, external-secret-backed Gmail credential storage on the primary path, refresh-token renewal through that seam, rotated refresh-token persistence through that seam, and the narrow `legacy_db_v0` transition path for older credential rows. - Use [PRODUCT_BRIEF.md](/Users/samirusani/Desktop/Codex/AliceBot/PRODUCT_BRIEF.md) for product scope, [ARCHITECTURE.md](/Users/samirusani/Desktop/Codex/AliceBot/ARCHITECTURE.md) for implemented technical boundaries, [ROADMAP.md](/Users/samirusani/Desktop/Codex/AliceBot/ROADMAP.md) for forward planning, and [RULES.md](/Users/samirusani/Desktop/Codex/AliceBot/RULES.md) for durable operating rules. - Historical build and review reports remain the source of sprint-by-sprint detail; the active handoff should stay compact and current. ## Implemented Repo Slice -- `apps/api` is the only shipped product surface. It implements continuity, tracing, deterministic context compilation, governed memory admission and review, embeddings, semantic retrieval, entities, policy and tool governance, approval persistence and resolution, approved-only `proxy.echo` execution, execution budgets, task/task-step lifecycle reads and mutations, explicit manual continuation lineage, explicit task-step linkage for approval and execution synchronization, deterministic rooted local task-workspace provisioning, explicit task-artifact registration, narrow local text/PDF/DOCX/RFC822 artifact ingestion into durable chunk rows, artifact-chunk embeddings, direct lexical and semantic artifact retrieval, compile-path semantic artifact retrieval, deterministic hybrid lexical-plus-semantic artifact merge inside the compile response, and a narrow read-only Gmail seam with protected credential storage plus single-message ingestion into the existing RFC822 artifact path. +- `apps/api` is the only shipped product surface. It implements continuity, tracing, deterministic context compilation, governed memory admission and review, embeddings, semantic retrieval, entities, policy and tool governance, approval persistence and resolution, approved-only `proxy.echo` execution, execution budgets, task/task-step lifecycle reads and mutations, explicit manual continuation lineage, explicit task-step linkage for approval and execution synchronization, deterministic rooted local task-workspace provisioning, explicit task-artifact registration, narrow local text/PDF/DOCX/RFC822 artifact ingestion into durable chunk rows, artifact-chunk embeddings, direct lexical and semantic artifact retrieval, compile-path semantic artifact retrieval, deterministic hybrid lexical-plus-semantic artifact merge inside the compile response, and a narrow read-only Gmail seam with external-secret-backed primary credentials plus single-message ingestion into the existing RFC822 artifact path. - The live schema includes continuity, trace, memory, embedding, entity, governance, `tasks`, `task_steps`, `task_workspaces`, `task_artifacts`, `task_artifact_chunks`, and `task_artifact_chunk_embeddings` tables with row-level security on user-owned data. -- The live schema also includes `gmail_accounts` and `gmail_account_credentials` for the shipped Gmail seam; account metadata reads stay secret-free while protected credentials remain isolated to the credential table. +- The live schema also includes `gmail_accounts` and `gmail_account_credentials` for the shipped Gmail seam; account metadata reads stay secret-free while `gmail_account_credentials` now stores locator metadata on the primary path and keeps the narrow `legacy_db_v0` transition state explicit for older rows. - `apps/web` and `workers` remain starter scaffolds only. ## Current Boundaries @@ -20,14 +20,14 @@ - Artifact retrieval operates only over persisted chunk rows and persisted chunk embeddings for one visible task or one visible artifact at a time; compile does not read raw files directly. - Compile can now include artifact chunks from lexical retrieval, semantic retrieval, or a deterministic hybrid merge of both into one artifact section with explicit per-chunk source provenance. - The shipped multi-step task path is still explicit and narrow: later steps are appended manually with lineage, while approval and execution synchronization use explicit linked `task_step_id` references. -- The shipped Gmail path is still explicit and narrow: one read-only account seam, one selected-message ingestion path, secret-free account reads, protected credential renewal for expired refresh-capable credentials, and rotated refresh-token persistence only inside `gmail_account_credentials`. +- The shipped Gmail path is still explicit and narrow: one read-only account seam, one selected-message ingestion path, secret-free account reads, one explicit secret-manager seam for primary credential reads and writes, refresh-capable credential renewal through that seam, rotated refresh-token persistence through that seam, and one explicit `legacy_db_v0` first-read externalization path for older rows. - The only execution handler in the repo is the in-process no-external-I/O `proxy.echo` path. ## Not Implemented - Rich document parsing beyond the shipped narrow local text/PDF/DOCX/RFC822 ingestion seam. - Gmail search, mailbox sync, attachment ingestion, write-capable Gmail actions, and Calendar connectors. -- External secret-manager integration for Gmail protected credentials. +- Removal of the remaining `legacy_db_v0` transition path for older Gmail credential rows. - Runner-style orchestration or automatic multi-step progression. - Artifact reranking or weighted fusion beyond the current lexical-first hybrid compile merge. - Auth beyond the current database user-context model. @@ -36,17 +36,17 @@ - Memory extraction and retrieval quality remain the main product risk. - Auth is still incomplete beyond database user context. -- Artifact ingestion and retrieval are intentionally narrow and local; broader document parsing, broader Gmail scope, external secret storage, and any retrieval changes beyond the shipped hybrid compile contract still need their own accepted seams. +- Artifact ingestion and retrieval are intentionally narrow and local; broader document parsing, broader Gmail scope, `legacy_db_v0` cleanup, and any retrieval changes beyond the shipped hybrid compile contract still need their own accepted seams. ## Latest Accepted Verification -- Latest accepted runtime verification totals for the shipped Sprint 5R seams were: - - `./.venv/bin/python -m pytest tests/unit` -> `437 passed` - - `./.venv/bin/python -m pytest tests/integration` -> `139 passed` -- Sprint 5S is documentation-only truth synchronization; it does not change runtime, schema, or API behavior. +- Latest accepted runtime verification totals for the shipped Sprint 5T seams were: + - `./.venv/bin/python -m pytest tests/unit` -> `446 passed` + - `./.venv/bin/python -m pytest tests/integration` -> `141 passed` +- Sprint 5U is documentation-only truth synchronization; it does not change runtime, schema, or API behavior. ## Planning Guardrails -- Plan from the implemented Sprint 5R repo state, not from older milestone narratives. -- Do not describe broader Gmail scope, Calendar work, runner work, UI work, external secret-manager integration, or artifact reranking beyond the current lexical-first hybrid compile merge as shipped. -- The immediate next move after this truth-sync sprint is one more narrow Gmail auth-adjacent sprint, most likely external secret-manager integration for the existing protected credential seam, without combining it with search, sync, Calendar, or UI expansion. +- Plan from the implemented Sprint 5T repo state, not from older milestone narratives. +- Do not describe broader Gmail scope, Calendar work, runner work, UI work, `legacy_db_v0` cleanup, or artifact reranking beyond the current lexical-first hybrid compile merge as shipped. +- The immediate next move after this truth-sync sprint is one more narrow Gmail auth-adjacent sprint, most likely removal of the remaining `legacy_db_v0` transition path for the existing credential seam, without combining it with search, sync, Calendar, or UI expansion. diff --git a/BUILD_REPORT.md b/BUILD_REPORT.md index 5320d7d..785d4a2 100644 --- a/BUILD_REPORT.md +++ b/BUILD_REPORT.md @@ -2,191 +2,76 @@ ## sprint objective -Implement Sprint 5T: externalize the Gmail protected-credential seam behind one explicit secret-manager adapter so `gmail_account_credentials` stores only non-secret locator metadata on the primary path, while preserving secret-free Gmail account reads and the existing single-message ingestion plus refresh/rotation flow. +Implement Sprint 5U: synchronize the live truth artifacts with the accepted repo state through Sprint 5T so planning and handoff docs accurately describe the shipped Gmail externalization seam and its remaining narrow transition boundary. ## completed work -- Added `GMAIL_SECRET_MANAGER_URL` runtime config as the single explicit runtime selector for the Gmail secret-manager adapter. Runtime no longer falls back silently when it is unset. -- Added a Gmail-only external secret-manager adapter in `apps/api/src/alicebot_api/gmail_secret_manager.py`. -- Updated Gmail credential persistence so new connect writes store secrets in the configured file-backed secret manager and persist only: - - `auth_kind` - - `credential_kind` - - `secret_manager_kind` - - `secret_ref` - - `credential_blob = NULL` -- Added migration `20260316_0029_gmail_external_secret_manager.py` to: - - add `credential_kind`, `secret_manager_kind`, and `secret_ref` to `gmail_account_credentials` - - make `credential_blob` nullable - - mark pre-existing rows as `secret_manager_kind = 'legacy_db_v0'` - - enforce that externalized rows are reference-only and legacy rows remain explicitly transitional -- Updated the Gmail service path so: - - connect writes secrets through the adapter - - ingestion resolves secrets through the adapter - - expired-token refresh writes updated secrets through the adapter - - rotated refresh tokens persist through the adapter - - secret-resolution and secret-update failures return deterministic Gmail errors without corrupting task artifact state -- Preserved Gmail account list/detail response shape and kept those responses secret-free. -- Updated `ARCHITECTURE.md` so the implemented slice and Gmail credential description now reflect Sprint 5T. -- Restored `.ai/active/SPRINT_PACKET.md` to the Control Tower-owned version after removing the unintended builder edit. -- Added unit and integration coverage for: - - external secret reference persistence - - secret-free account responses and normal table reads - - ingestion through the externalized path - - refresh and rotation through the externalized path - - missing external secret failure handling - - legacy-row transition behavior - -## exact Gmail credential schema and contract changes introduced - -- Schema: - - `apps/api/alembic/versions/20260316_0029_gmail_external_secret_manager.py` - - `gmail_account_credentials` now includes: - - `credential_kind text not null` - - `secret_manager_kind text not null` - - `secret_ref text null` - - `credential_blob jsonb null` - - Storage rule: - - externalized rows: `secret_manager_kind = 'file_v1'`, non-empty `secret_ref`, `credential_blob IS NULL` - - transition rows: `secret_manager_kind = 'legacy_db_v0'`, `secret_ref IS NULL`, `credential_blob` still contains the old protected payload until first credential read externalizes it -- Internal contract changes: - - `Settings` now includes `gmail_secret_manager_url` - - Gmail store rows and write/update methods now carry `credential_kind`, `secret_manager_kind`, and `secret_ref` -- Public API contracts: - - no Gmail account read response fields changed - - no Gmail ingestion response fields changed - - no request payload shape changed - -## external secret-manager adapter rule used - -- Adapter selector: `GMAIL_SECRET_MANAGER_URL` -- Supported rule in this sprint: `file://` -- Runtime requirement when unset: fail fast and require `GMAIL_SECRET_MANAGER_URL` to be configured explicitly -- Deterministic local fallback remains a test-only implementation detail via explicit test configuration of `file://` -- Secret reference format persisted in `gmail_account_credentials.secret_ref`: - - `users//gmail-account-credentials/.json` -- Secret payload format stored outside the database: - - same Gmail credential object previously stored in `credential_blob` - - includes `credential_kind`, `access_token`, and refresh metadata when present +- Audited the shipped Sprint 5T repo state against the current truth docs. +- Confirmed `ARCHITECTURE.md` already matched the accepted Sprint 5T implementation, so no architecture correction was required. +- Updated `ROADMAP.md` from a Sprint 5R baseline to a Sprint 5T baseline. +- Updated `.ai/handoff/CURRENT_STATE.md` from a Sprint 5R baseline to a Sprint 5T baseline. +- Replaced stale forward-looking language that still treated Gmail external secret-manager integration as future work. +- Reframed the immediate next narrow Gmail seam as `legacy_db_v0` cleanup rather than external-secret-manager integration. +- Preserved all out-of-scope boundaries: no runtime, schema, API, connector-breadth, runner, or UI changes. ## incomplete work -- None inside Sprint 5T scope. +- None inside Sprint 5U scope. ## files changed -- `apps/api/src/alicebot_api/config.py` -- `apps/api/src/alicebot_api/gmail.py` -- `apps/api/src/alicebot_api/gmail_secret_manager.py` -- `apps/api/src/alicebot_api/main.py` -- `apps/api/src/alicebot_api/store.py` -- `apps/api/alembic/versions/20260316_0029_gmail_external_secret_manager.py` -- `tests/unit/test_config.py` -- `tests/unit/test_gmail.py` -- `tests/unit/test_gmail_main.py` -- `tests/unit/test_gmail_secret_manager.py` -- `tests/unit/test_20260316_0029_gmail_external_secret_manager.py` -- `tests/integration/test_gmail_accounts_api.py` -- `tests/integration/test_migrations.py` -- `ARCHITECTURE.md` +- `ROADMAP.md` +- `.ai/handoff/CURRENT_STATE.md` - `BUILD_REPORT.md` ## tests run -- `./.venv/bin/python -m pytest tests/unit/test_config.py tests/unit/test_gmail_secret_manager.py tests/unit/test_gmail.py tests/unit/test_gmail_main.py tests/unit/test_20260316_0029_gmail_external_secret_manager.py` - - Result: `38 passed` -- `./.venv/bin/python -m pytest tests/unit` - - Result: `446 passed` -- `./.venv/bin/python -m pytest tests/integration/test_migrations.py tests/integration/test_gmail_accounts_api.py` - - Result: `16 passed` -- `./.venv/bin/python -m pytest tests/integration` - - Result: `141 passed` -- `./.venv/bin/python -m pytest tests/unit` - - Result after review fixes: `446 passed` -- `./.venv/bin/python -m pytest tests/integration` - - Result after review fixes: `141 passed` -- `git diff --check -- apps/api/src/alicebot_api/config.py apps/api/src/alicebot_api/gmail.py apps/api/src/alicebot_api/main.py apps/api/src/alicebot_api/store.py apps/api/src/alicebot_api/gmail_secret_manager.py apps/api/alembic/versions/20260316_0029_gmail_external_secret_manager.py tests/unit/test_config.py tests/unit/test_gmail.py tests/unit/test_gmail_main.py tests/unit/test_gmail_secret_manager.py tests/unit/test_20260316_0029_gmail_external_secret_manager.py tests/integration/test_gmail_accounts_api.py tests/integration/test_migrations.py` - - Result: no diff formatting errors - -## one example Gmail account response proving secret-free reads remain intact - -```json -{ - "account": { - "id": "00000000-0000-0000-0000-000000000001", - "provider": "gmail", - "auth_kind": "oauth_access_token", - "provider_account_id": "acct-owner-001", - "email_address": "owner@gmail.example", - "display_name": "Owner", - "scope": "https://www.googleapis.com/auth/gmail.readonly", - "created_at": "2026-03-16T10:00:00+00:00", - "updated_at": "2026-03-16T10:00:00+00:00" - } -} -``` - -- No `access_token`, `refresh_token`, `client_id`, or `client_secret` fields are present on the read surface. - -## one example Gmail ingestion response through the externalized credential path - -```json -{ - "account": { - "id": "00000000-0000-0000-0000-000000000001", - "provider": "gmail", - "auth_kind": "oauth_access_token", - "provider_account_id": "acct-owner-refresh-001", - "email_address": "owner@gmail.example", - "display_name": "Owner", - "scope": "https://www.googleapis.com/auth/gmail.readonly", - "created_at": "2026-03-16T10:00:00+00:00", - "updated_at": "2026-03-16T10:00:00+00:00" - }, - "message": { - "provider_message_id": "msg-001", - "artifact_relative_path": "gmail/acct-owner-refresh-001/msg-001.eml", - "media_type": "message/rfc822" - }, - "artifact": { - "id": "00000000-0000-0000-0000-000000000123", - "task_id": "00000000-0000-0000-0000-000000000010", - "task_workspace_id": "00000000-0000-0000-0000-000000000020", - "status": "registered", - "ingestion_status": "ingested", - "relative_path": "gmail/acct-owner-refresh-001/msg-001.eml", - "media_type_hint": "message/rfc822", - "created_at": "2026-03-16T10:00:00+00:00", - "updated_at": "2026-03-16T10:00:01+00:00" - }, - "summary": { - "total_count": 1, - "total_characters": 16, - "media_type": "message/rfc822", - "chunking_rule": "normalized_utf8_text_fixed_window_1000_chars_v1", - "order": ["sequence_no_asc", "id_asc"] - } -} -``` +- `git diff --check -- ROADMAP.md .ai/handoff/CURRENT_STATE.md BUILD_REPORT.md` ## blockers/issues -- No implementation blockers remained. -- Full integration verification required local Postgres access outside the default sandbox; the escalated run completed successfully. -- Transitional note: pre-Sprint-5T `gmail_account_credentials` rows are marked `legacy_db_v0` by migration and externalize on first credential read instead of through a bulk secret-export migration. +- No implementation blockers occurred. +- `ARCHITECTURE.md` was audited but not edited because it already reflected the accepted Sprint 5T state. +- The worktree already contained unrelated changes outside sprint scope; they were left untouched. -## what remains intentionally deferred to later milestones +## accepted evidence used -- Gmail search -- mailbox sync or backfill jobs -- attachment ingestion -- write-capable Gmail actions -- Calendar connector scope -- OAuth UI or callback handling -- broader cross-provider secret abstraction -- compile-contract changes -- runner-style orchestration -- UI work +- `ARCHITECTURE.md` current Sprint 5T implemented-slice and Gmail boundary text. +- `BUILD_REPORT.md` from Sprint 5T, which records the accepted external secret-manager seam, locator-only primary credential storage, and the `legacy_db_v0` transition path. +- `REVIEW_REPORT.md` with `PASS` for Sprint 5T and accepted verification totals: + - `./.venv/bin/python -m pytest tests/unit` -> `446 passed` + - `./.venv/bin/python -m pytest tests/integration` -> `141 passed` +- Repo implementation and test evidence in: + - `apps/api/src/alicebot_api/gmail.py` + - `apps/api/src/alicebot_api/gmail_secret_manager.py` + - `tests/integration/test_gmail_accounts_api.py` + - `tests/integration/test_migrations.py` + +## specific stale statements corrected + +- `ROADMAP.md` no longer says the repo is current only through Sprint 5R. +- `ROADMAP.md` no longer says external secret-manager integration is the next Gmail auth seam. +- `.ai/handoff/CURRENT_STATE.md` no longer says the repo is current only through Sprint 5R. +- `.ai/handoff/CURRENT_STATE.md` no longer lists external secret-manager integration as not implemented. +- `.ai/handoff/CURRENT_STATE.md` no longer points planning at external secret-manager integration as the immediate next move. +- `.ai/handoff/CURRENT_STATE.md` now carries the accepted Sprint 5T verification totals instead of the older Sprint 5R totals. + +## confirmation of non-runtime scope + +- No runtime code changed. +- No schema or migration files changed. +- No API contract changed. +- No Gmail connector breadth changed. +- No runner or UI scope entered the diff. + +## what remains intentionally deferred after truth synchronization + +- Removal of the remaining `legacy_db_v0` transition path for older Gmail credential rows. +- Gmail search, mailbox sync, attachment ingestion, and write-capable Gmail actions. +- Calendar connector scope. +- Richer document parsing beyond the current narrow local ingestion seams. +- Runner-style orchestration and UI work. ## recommended next step -Keep the next sprint narrow around one follow-up seam only: either remove the remaining `legacy_db_v0` transition path with a deliberate migration/export plan, or move to the next Gmail behavior slice without widening into search, sync, Calendar, runner, or UI work. +Open one narrow follow-up sprint to remove the remaining `legacy_db_v0` transition path deliberately, without widening into Gmail search, sync, attachments, Calendar, runner, or UI scope. diff --git a/REVIEW_REPORT.md b/REVIEW_REPORT.md index ccfa070..fc4acf5 100644 --- a/REVIEW_REPORT.md +++ b/REVIEW_REPORT.md @@ -6,20 +6,13 @@ PASS ## criteria met -- The active control artifact now matches the implemented sprint: - - `.ai/active/SPRINT_PACKET.md` is Sprint 5T - - branch name matches `codex/sprint-5t-gmail-external-secret-manager` -- The primary Gmail protected credential path no longer depends on application-table-stored secret material for new writes. -- `gmail_account_credentials` persists only non-secret locator/reference metadata for the externalized primary path, while secret payloads are stored through one explicit Gmail secret-manager adapter boundary. -- Gmail account list and detail responses remain secret-free. -- The existing single-message Gmail ingestion path still works through the external secret-manager seam. -- Refresh-token renewal and rotated refresh-token persistence still work through the externalized seam. -- Secret-resolution and secret-update failures are deterministic and test-backed. -- Per-user isolation and stable response shape are test-backed. -- No hidden Gmail search, sync, attachments, write actions, Calendar, compile-contract, runner, or UI scope entered the sprint. -- Verification confirmed in review: - - `./.venv/bin/python -m pytest tests/unit` -> `446 passed` - - `./.venv/bin/python -m pytest tests/integration` -> `141 passed` +- The sprint stayed documentation-only. The diff changes only `.ai/active/SPRINT_PACKET.md`, `ROADMAP.md`, `.ai/handoff/CURRENT_STATE.md`, and `BUILD_REPORT.md`; no runtime, schema, API, connector-breadth, runner, or UI files changed. +- `ROADMAP.md` now reflects the accepted repo state through Sprint 5T instead of Sprint 5R. +- `ROADMAP.md` no longer describes external secret-manager integration as future work and now correctly frames the next narrow seam as `legacy_db_v0` cleanup. +- `.ai/handoff/CURRENT_STATE.md` now reflects the implemented Sprint 5T Gmail seam: external-secret-backed primary credential storage, secret-free reads, renewal through the externalized seam, rotated refresh-token persistence, and the narrow `legacy_db_v0` transition path. +- The updated truth artifacts distinguish implemented Gmail externalization behavior from deferred Gmail breadth and from the still-deferred `legacy_db_v0` cleanup sprint. +- `BUILD_REPORT.md` identifies the synchronized truth artifacts, cites accepted Sprint 5T evidence, names the stale statements corrected, confirms the non-runtime scope, and states what remains deferred. +- `ARCHITECTURE.md` already matched the accepted Sprint 5T implementation and did not require edits; the unchanged text remains consistent with the updated roadmap and handoff documents. ## criteria missed @@ -27,30 +20,31 @@ PASS ## quality issues -- No blocking implementation-quality issues remain for Sprint 5T. -- The prior control-artifact mismatch was resolved before this review pass; packet, branch, implementation, and build report are now aligned. +- No blocking quality issues found in the Sprint 5U diff. +- The builder kept the sprint narrow and did not overreach beyond the truth-sync scope. ## regression risks - Low. -- The only notable operational caution is the intentional `legacy_db_v0` transition path, which is narrow, explicit, and covered by tests, but should still be removed deliberately in a later sprint rather than allowed to linger indefinitely. +- The only meaningful residual risk is planning drift if future truth-sync sprints again update `ROADMAP.md` and `.ai/handoff/CURRENT_STATE.md` without checking them against the accepted implementation and accepted reports. +- Runtime risk from this sprint is effectively nil because no runtime files changed. ## docs issues -- None blocking. -- `ARCHITECTURE.md` now reflects Sprint 5T and the externalized Gmail credential seam. -- `BUILD_REPORT.md` now matches the Sprint 5T packet and the corrected runtime contract. +- No blocking docs issues. +- Minor note only: the evidence cited in `BUILD_REPORT.md` depends partly on accepted Sprint 5T artifacts now visible through git history rather than through separate preserved files in the working tree. That is still adequate for this sprint and does not block acceptance. ## should anything be added to RULES.md? - No. -- The existing rules were sufficient; the earlier issue was drift in the active control artifact, not a missing repo rule. +- The repo already had sufficient scope-control rules. The problem addressed here was stale truth artifacts, not a missing durable rule. ## should anything update ARCHITECTURE.md? -- No further update is required for this sprint beyond the changes already made. +- No. +- The current `ARCHITECTURE.md` already describes the accepted Sprint 5T Gmail seam accurately, including the external secret-manager boundary and the narrow `legacy_db_v0` transition path. ## recommended next action -- Accept Sprint 5T. -- Keep the next sprint narrow, most likely around removing the `legacy_db_v0` transition path with an explicit migration/export plan or moving to the next Gmail behavior slice without widening into search, sync, Calendar, runner, or UI scope. +- Accept Sprint 5U. +- Open one narrow follow-up sprint to remove the remaining `legacy_db_v0` transition path without widening into Gmail search, sync, attachments, Calendar, runner, or UI scope. diff --git a/ROADMAP.md b/ROADMAP.md index 936fca2..e1a83f4 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -2,18 +2,18 @@ ## Current Position -- The accepted repo state is current through Sprint 5R. +- The accepted repo state is current through Sprint 5T. - Milestone 5 now ships the rooted local workspace and artifact baseline end to end: workspace provisioning, artifact registration, narrow text ingestion, narrow PDF/DOCX/RFC822 ingestion, durable chunk storage, lexical artifact retrieval, compile-path artifact inclusion, artifact-chunk embeddings, direct semantic artifact retrieval, compile-path semantic artifact retrieval, and deterministic hybrid lexical-plus-semantic artifact merge in compile. -- The same milestone also now ships the narrow Gmail seam: read-only Gmail account persistence, secret-free account reads, protected credential storage in `gmail_account_credentials`, refresh-token renewal for expired access tokens, rotated refresh-token persistence when the provider returns a replacement token, and one explicit selected-message ingestion path that lands in the existing RFC822 artifact pipeline. +- The same milestone also now ships the narrow Gmail seam: read-only Gmail account persistence, secret-free account reads, external-secret-backed primary credential storage with locator metadata in `gmail_account_credentials`, refresh-token renewal for expired access tokens through the externalized seam, rotated refresh-token persistence when the provider returns a replacement token, one narrow `legacy_db_v0` transition path for older rows, and one explicit selected-message ingestion path that lands in the existing RFC822 artifact pipeline. - This roadmap is future-facing from that shipped baseline; historical sprint-by-sprint detail lives in accepted build and review artifacts, not here. ## Next Delivery Focus -### Open One More Narrow Gmail Auth Seam On Top Of The Shipped Baseline +### Remove The Remaining Narrow Gmail Transition Path Without Widening Scope -- Keep the next sprint auth-adjacent and narrow, building on the shipped protected-credential-backed Gmail seam rather than widening connector breadth. -- The next best seam is external secret-manager integration for the existing `gmail_account_credentials` boundary, without changing the read-only account contract or the single-message ingestion contract. -- Do not combine that work with Gmail search, mailbox sync, attachment ingestion, Calendar scope, UI work, or broader connector orchestration. +- Keep the next sprint auth-adjacent and narrow, building on the shipped external-secret-backed Gmail seam rather than widening connector breadth. +- The next best seam is deliberate cleanup of the remaining `legacy_db_v0` transition boundary for older `gmail_account_credentials` rows, without changing the read-only account contract or the single-message ingestion contract. +- Do not combine that cleanup with Gmail search, mailbox sync, attachment ingestion, Calendar scope, UI work, or broader connector orchestration. ### Preserve Current Document, Compile, Governance, And Task Guarantees @@ -24,7 +24,7 @@ ## After The Next Narrow Sprint -- Reassess broader connector work only after the current Gmail protected-credential boundary remains stable under externalized secret storage and the truth artifacts stay synchronized. +- Reassess broader connector work only after the current Gmail external-secret-backed credential boundary remains stable without the `legacy_db_v0` transition path and the truth artifacts stay synchronized. - Revisit workflow UI only after backend document and connector seams are accepted and the truth artifacts stay current. - Revisit broader task orchestration only after the current explicit task-step seams remain stable under workspace, artifact, document, and connector flows. - Continue to defer broader tool execution breadth and production auth/deployment hardening until the current governed surface remains stable. @@ -33,11 +33,11 @@ - Live truth docs must stay synchronized with accepted repo state so sprint planning does not start from stale assumptions. - Rich document parsing work should continue to build on the shipped rooted local workspace, durable artifact chunk, and hybrid compile retrieval contracts. -- Connector work should remain read-only, single-message-only, approval-aware, and protected-credential-backed until a later sprint explicitly opens broader scope. +- Connector work should remain read-only, single-message-only, approval-aware, and external-secret-backed until a later sprint explicitly opens broader scope. - Runner-style orchestration should stay deferred until the repo no longer depends on narrow current-step assumptions for safety and explainability. ## Ongoing Risks - Memory extraction and retrieval quality remain the largest product risk. - Auth beyond database user context is still missing. -- Milestone 5 can drift if Gmail auth hardening, broader connector breadth, UI, richer parsing, and orchestration work are mixed into one sprint instead of landing as narrow seams on top of the shipped document-ingestion and Gmail baseline. +- Milestone 5 can drift if `legacy_db_v0` cleanup, broader Gmail breadth, UI, richer parsing, and orchestration work are mixed into one sprint instead of landing as narrow seams on top of the shipped document-ingestion and Gmail externalization baseline.