feat(search-engine): add Elasticsearch backend and canonical import/sync#44#45
Merged
Intrinsical-AI merged 21 commits intodevelopfrom Mar 14, 2026
Merged
feat(search-engine): add Elasticsearch backend and canonical import/sync#44#45Intrinsical-AI merged 21 commits intodevelopfrom
Intrinsical-AI merged 21 commits intodevelopfrom
Conversation
Add nullable scope and snapshot_id fields to the SQL Document table and Elasticsearch index mappings, enabling external producers to tag documents with an origin scope and version snapshot.
…ields Extend UpsertDocBuilderPort, MutationUpsertInput, and normalization/serialization to carry scope and snapshot_id. Add CanonicalImportSummary result type for reporting insert/update/delete statistics after a canonical import.
…xecutors Wire the new fields from MutationUpsertInput into the upsert builder calls in both the atomic and saga mutation execution paths.
…nal_ids_by_scope() Update SQL and Elasticsearch repositories to read/write scope and snapshot_id in upsert, change detection, and domain mapping. Add list_external_ids_by_scope() to both stores to support stale-doc deletion in canonical imports.
Introduce CanonicalImportRequest/Response schemas with scope, snapshot_id, replace_scope flag, and duplicate external_id validation. Add the endpoint behind the multi-store write lock. Extend existing mutate endpoint schemas to accept scope/snapshot_id on individual upsert items.
Implement execute_import_canonical_sync() use case: batched upsert (256 docs) with optional replace_scope hard-deletion of stale documents not present in the current snapshot. Wire as rag-import-canonical CLI entry point and register in the CLI group.
Document the new import-canonical HTTP endpoint and CLI command with curl and CLI invocation examples. Note scope/snapshot synchronization semantics and replace_scope behavior.
…l endpoints (#47) * refactor(test-suite): update, expand, polish. Better conceptual splitting, polished fixtures/conftests. Added tests for critical endpoints * [waterfall-pr-chain] feat(filter): metadata based filtering. (#48) feat(filter): metadata based filtering. Expander contract to accept metadata.<key> plus legacy fields. Updated Elastic/OpenSearch + Solr to exact resolve filtering
Owner
Author
|
UPDATE: aggregated PRs included:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
system state, diagnostics, and index mapping management; wired through
composition and runtime alongside the existing SQL backend.
mutation contracts, both persistence adapters (SQL + ES), HTTP schemas,
and CLI — enabling documents to be tagged with an origin scope and version.
POST /api/docs/import-canonical) andCLI command (
rag-import-canonical): declarative snapshot-based syncthat upserts a scoped document collection and optionally hard-deletes stale
docs not present in the current snapshot (
replace_scopemode).Test plan
list_external_ids_by_scope)rag-import-canonicalCLI (sync + cross-snapshot deletion)import-canonicalcommand and entry point are wireddocker-compose up, index docs viarag-import-canonical, verify scope-aware deletion