feat(search-engine): add Elasticsearch backend and canonical import/sync#44
Closed
Intrinsical-AI wants to merge 13 commits intomasterfrom
Closed
feat(search-engine): add Elasticsearch backend and canonical import/sync#44Intrinsical-AI wants to merge 13 commits intomasterfrom
Intrinsical-AI wants to merge 13 commits intomasterfrom
Conversation
Add nullable scope and snapshot_id fields to the SQL Document table and Elasticsearch index mappings, enabling external producers to tag documents with an origin scope and version snapshot.
…ields Extend UpsertDocBuilderPort, MutationUpsertInput, and normalization/serialization to carry scope and snapshot_id. Add CanonicalImportSummary result type for reporting insert/update/delete statistics after a canonical import.
…xecutors Wire the new fields from MutationUpsertInput into the upsert builder calls in both the atomic and saga mutation execution paths.
…nal_ids_by_scope() Update SQL and Elasticsearch repositories to read/write scope and snapshot_id in upsert, change detection, and domain mapping. Add list_external_ids_by_scope() to both stores to support stale-doc deletion in canonical imports.
Introduce CanonicalImportRequest/Response schemas with scope, snapshot_id, replace_scope flag, and duplicate external_id validation. Add the endpoint behind the multi-store write lock. Extend existing mutate endpoint schemas to accept scope/snapshot_id on individual upsert items.
Implement execute_import_canonical_sync() use case: batched upsert (256 docs) with optional replace_scope hard-deletion of stale documents not present in the current snapshot. Wire as rag-import-canonical CLI entry point and register in the CLI group.
Document the new import-canonical HTTP endpoint and CLI command with curl and CLI invocation examples. Note scope/snapshot synchronization semantics and replace_scope behavior.
Owner
Author
|
Unstable - mergin to dev first |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
system state, diagnostics, and index mapping management; wired through
composition and runtime alongside the existing SQL backend.
mutation contracts, both persistence adapters (SQL + ES), HTTP schemas,
and CLI — enabling documents to be tagged with an origin scope and version.
POST /api/docs/import-canonical) andCLI command (
rag-import-canonical): declarative snapshot-based syncthat upserts a scoped document collection and optionally hard-deletes stale
docs not present in the current snapshot (
replace_scopemode).Test plan
list_external_ids_by_scope)rag-import-canonicalCLI (sync + cross-snapshot deletion)import-canonicalcommand and entry point are wireddocker-compose up, index docs viarag-import-canonical, verify scope-aware deletion