fix: Filter chat scope#1228
Conversation
- Add utils.opensearch_filter_merge (coerce clauses, merge into body, limits, apply_chat_filter_expression_to_search_body) with unit tests - Wire Langflow OpenSearch component raw_search and _coerce_filter_clauses to utils; consistent ValueError for invalid query/filter JSON - Dockerfile.langflow: COPY src and PYTHONPATH for utils import - Sync embedded flow JSON from component source Made-with: Cursor
Top-level knn blocks do not inherit query.bool.filter; scope filters into each vector field kNN filter and skip filter-only query when kNN-only body. Made-with: Cursor
github-actions
Bot
added
backend
There was a problem hiding this comment.
Functional Review 1
- ✅ LGTM / Approved
- ✅ Latest changes on the filter-chat-scope branch appears to rectify the issue
- For details, see: #1130 (comment)
- Note: No Code Review performed (deferring to the other reviews for this aspect)
|
merging this PR with functional review only because it needs to be tested by QA tomorrow |
| from opensearchpy import OpenSearch, helpers | ||
| from opensearchpy.exceptions import OpenSearchException, RequestError | ||
|
|
||
| from utils.opensearch_filter_merge import ( |
There was a problem hiding this comment.
This isa Langflwo component code.
Hence importing packages from the OpenRAG folder would not work
There was a problem hiding this comment.
Valid concern. It is addressed this by updating Dockerfile.langflow to copy OpenRAG src into the Langflow image and set PYTHONPATH=/app/openrag-src. So the component can import utils.opensearch_filter_merge when running this branch’s image. Without that Dockerfile change (older image), the import would fail.
There was a problem hiding this comment.
True but this would bring inconsistency in the apporach and we should device an alternative fix.
3 participants
issue: #1130
What changed
src/utils/opensearch_filter_merge.py
Centralizes coercion of chat filter JSON into OpenSearch term / terms clauses (explicit filter array + context keys aligned with chat_service, including connector_types → connector_type).
merge_filter_clauses_into_search_body: merges clauses into query via bool (existing behavior), and scopes top-level knn by injecting filters into each vector field’s filter (OpenSearch does not apply query.bool.filter to sibling top-level knn). Avoids adding a filter-only query when the body is kNN-only in a way that left kNN global.
Applies limit / score_threshold when size / min_score are not already set.
flows/components/opensearch_multimodal.py (+ synced flows/*.json)
raw_search runs the request body through apply_chat_filter_expression_to_search_body; stricter JSON validation with clear ValueErrors.
_coerce_filter_clauses delegates to the shared helper.
Dockerfile.langflow
COPY src and PYTHONPATH so from utils.opensearch_filter_merge import … works in the Langflow container.
docker-compose.yml / .env.example / Makefile / .gitignore / docs
Langflow DB/config dir under ./langflow-data, ensure-langflow-data for dev targets, factory-reset / CI cleanup include langflow-data, configuration docs.
tests/unit/test_opensearch_filter_merge.py
Unit coverage for merge/coerce behavior, including top-level knn.
How to verify
uv run pytest tests/unit/test_opensearch_filter_merge.py -v
Run stack with a locally built Langflow image if needed (docker compose build langflow or make dev-local-build-lf), so the Dockerfile PYTHONPATH + src copy match this branch.
In chat with a knowledge filter, confirm raw_search logs show query (after chat filter merge): … and results stay within selected sources.