Timestamp field bound precision#15
Closed
rdettai-sk wants to merge 113 commits intoSekoiaLab:mainfrom
Closed
Conversation
Bumps the github-actions group with 8 updates in the / directory: | Package | From | To | | --- | --- | --- | | [actions/checkout](https://github.com/actions/checkout) | `5.0.0` | `5.0.1` | | [dtolnay/rust-toolchain](https://github.com/dtolnay/rust-toolchain) | `6d653acede28d24f02e3cd41383119e8b1b35921` | `0f44b27771c32bda9f458f75a1e241b09791b331` | | [taiki-e/install-action](https://github.com/taiki-e/install-action) | `2.62.40` | `2.62.52` | | [sarisia/actions-status-discord](https://github.com/sarisia/actions-status-discord) | `1.15.4` | `1.15.5` | | [actions/dependency-review-action](https://github.com/actions/dependency-review-action) | `45529485b5eb76184ced07362d2331fd9d26f03f` | `125b99508212ce1cc3076ad60f6bd63bf6d88a66` | | [docker/setup-qemu-action](https://github.com/docker/setup-qemu-action) | `3.6.0` | `3.7.0` | | [docker/metadata-action](https://github.com/docker/metadata-action) | `5.8.0` | `5.9.0` | | [github/codeql-action](https://github.com/github/codeql-action) | `4.31.0` | `4.31.3` | Updates `actions/checkout` from 5.0.0 to 5.0.1 - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@08c6903...93cb6ef) Updates `dtolnay/rust-toolchain` from 6d653acede28d24f02e3cd41383119e8b1b35921 to 0f44b27771c32bda9f458f75a1e241b09791b331 - [Release notes](https://github.com/dtolnay/rust-toolchain/releases) - [Commits](dtolnay/rust-toolchain@6d653ac...0f44b27) Updates `taiki-e/install-action` from 2.62.40 to 2.62.52 - [Release notes](https://github.com/taiki-e/install-action/releases) - [Changelog](https://github.com/taiki-e/install-action/blob/main/CHANGELOG.md) - [Commits](taiki-e/install-action@41ef8c6...537c30d) Updates `sarisia/actions-status-discord` from 1.15.4 to 1.15.5 - [Release notes](https://github.com/sarisia/actions-status-discord/releases) - [Commits](sarisia/actions-status-discord@11a0bfe...b8381b2) Updates `actions/dependency-review-action` from 45529485b5eb76184ced07362d2331fd9d26f03f to 125b99508212ce1cc3076ad60f6bd63bf6d88a66 - [Release notes](https://github.com/actions/dependency-review-action/releases) - [Commits](actions/dependency-review-action@4552948...125b995) Updates `docker/setup-qemu-action` from 3.6.0 to 3.7.0 - [Release notes](https://github.com/docker/setup-qemu-action/releases) - [Commits](docker/setup-qemu-action@2910929...c7c5346) Updates `docker/metadata-action` from 5.8.0 to 5.9.0 - [Release notes](https://github.com/docker/metadata-action/releases) - [Commits](docker/metadata-action@c1e5197...318604b) Updates `github/codeql-action` from 4.31.0 to 4.31.3 - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@4e94bd1...014f16e) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: 5.0.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: github-actions - dependency-name: dtolnay/rust-toolchain dependency-version: 0f44b27771c32bda9f458f75a1e241b09791b331 dependency-type: direct:production dependency-group: github-actions - dependency-name: taiki-e/install-action dependency-version: 2.62.52 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: github-actions - dependency-name: sarisia/actions-status-discord dependency-version: 1.15.5 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: github-actions - dependency-name: actions/dependency-review-action dependency-version: 125b99508212ce1cc3076ad60f6bd63bf6d88a66 dependency-type: direct:production dependency-group: github-actions - dependency-name: docker/setup-qemu-action dependency-version: 3.7.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: github-actions - dependency-name: docker/metadata-action dependency-version: 5.9.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: github-actions - dependency-name: github/codeql-action dependency-version: 4.31.3 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: github-actions ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps debian from `78d2f66` to `936abff`. --- updated-dependencies: - dependency-name: debian dependency-version: bookworm-slim dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: fulmicoton <paul.masurel@datadoghq.com>
Bumps debian from `936abff` to `b4aa902`. --- updated-dependencies: - dependency-name: debian dependency-version: bookworm-slim dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps the github-actions group with 4 updates: [actions/checkout](https://github.com/actions/checkout), [dtolnay/rust-toolchain](https://github.com/dtolnay/rust-toolchain), [taiki-e/install-action](https://github.com/taiki-e/install-action) and [github/codeql-action](https://github.com/github/codeql-action). Updates `actions/checkout` from 5.0.1 to 6.0.0 - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@93cb6ef...1af3b93) Updates `dtolnay/rust-toolchain` from 0f44b27771c32bda9f458f75a1e241b09791b331 to 0b1efabc08b657293548b77fb76cc02d26091c7e - [Release notes](https://github.com/dtolnay/rust-toolchain/releases) - [Commits](dtolnay/rust-toolchain@0f44b27...0b1efab) Updates `taiki-e/install-action` from 2.62.52 to 2.62.57 - [Release notes](https://github.com/taiki-e/install-action/releases) - [Changelog](https://github.com/taiki-e/install-action/blob/main/CHANGELOG.md) - [Commits](taiki-e/install-action@537c30d...763e332) Updates `github/codeql-action` from 4.31.3 to 4.31.5 - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@014f16e...fdbfb4d) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: 6.0.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions - dependency-name: dtolnay/rust-toolchain dependency-version: 0b1efabc08b657293548b77fb76cc02d26091c7e dependency-type: direct:production dependency-group: github-actions - dependency-name: taiki-e/install-action dependency-version: 2.62.57 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: github-actions - dependency-name: github/codeql-action dependency-version: 4.31.5 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: github-actions ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Bump tantivy version * Update 3rd part license file --------- Co-authored-by: fulmicoton <paul.masurel@datadoghq.com>
* [ui] remove unused asset * [ui] avoid using automatically generated svg component (for the sake of having less non-standard configuration) * [ui] replace react-monaco-editor with @monaco-editor/react which seems to be the officially supported one * [ui] use vite for build and local dev server * [ui] run type checking on CI * [ui] eject from react-create-app and fix jest config * [ui] ensure install step is determinist * [ui] update readme * [ui] fix linting errors + replace eslint by biome * [ui] bump typescript version * [ui] bump react 19 * [ui] Avoid using loader-utils@0.2.17 , glob@10.4.5 , dompurify@3.1.7 which are flag as vulnerable
* Skipping running tantivy search if cannot do better. Also, - removes some noisy intermediary traces/span. - logs the ast/agg in the root node. - logs a summary about the split execution (cache, pruned, etc.) in the leaf node Co-authored-by: fulmicoton <paul.masurel@datadoghq.com>
Bumps [pprof](https://github.com/tikv/pprof-rs) from 0.13.0 to 0.14.0. - [Changelog](https://github.com/tikv/pprof-rs/blob/master/CHANGELOG.md) - [Commits](https://github.com/tikv/pprof-rs/commits) --- updated-dependencies: - dependency-name: pprof dependency-version: 0.14.0 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* feat: Prefix and Wildcard in Elastic DSL Signed-off-by: Darkheir <raphael.cohen@sekoia.io> * chore: Add licence header Signed-off-by: Darkheir <raphael.cohen@sekoia.io> * feat: Add integration tests Signed-off-by: Darkheir <raphael.cohen@sekoia.io> * feat: Add support for short request format Signed-off-by: Darkheir <raphael.cohen@sekoia.io> * Apply PR review sugestion Signed-off-by: Darkheir <raphael.cohen@sekoia.io> --------- Signed-off-by: Darkheir <raphael.cohen@sekoia.io>
* feat: Add case_insensitive param to some Elastic DSL queries Signed-off-by: Darkheir <raphael.cohen@sekoia.io> * Apply PR review sugestion Signed-off-by: Darkheir <raphael.cohen@sekoia.io> --------- Signed-off-by: Darkheir <raphael.cohen@sekoia.io>
* update tantivy * allow deprecated * new TopDocs api
* introduce BuildTantivyAstContext * implement CacheNode * implement predicate cache * emit cachenode for search_after
* bump compatible deps * update first incompatible batch * add second incompatible batch * add third incompatible batch * update the updated function * updatr 3rd party license * just add disk feature from sysinfo * update 3rd license
* bump compatible deps * update first incompatible batch * add second incompatible batch * add third incompatible batch * update the updated function * updatr 3rd party license * bump more deps * remove tests * update tests * update 3rd license * fix tests * fix lint * just add disk feature from sysinfo * update 3rd license
* bump compatible deps * update first incompatible batch * add second incompatible batch * add third incompatible batch * update the updated function * updatr 3rd party license * bump more deps * remove tests * update tests * update 3rd license * fix tests * add more deps * fix lint * update 3rd license * just add disk feature from sysinfo * update 3rd license * fix fmt
* fix gcs tests * update the gcs tests * remove comments * add default impl * fix fmt * address comments * fix tests
The cause is complicated. When relying on localhost for connection, the client will first attempt ipv6 and then ipv4. This will cause a bunch of BrokerTransportFailure to be emitted by RdKafka. These errors will either fail the actor or fail the test assert. See #6049 Co-authored-by: fulmicoton <paul.masurel@datadoghq.com>
* Waiting for zookeeper to be up and running. * Removing zookeeper and lighter pulsar.
and increases the yarn download timeout Co-authored-by: fulmicoton <paul.masurel@datadoghq.com>
Co-authored-by: Pascal Seitz <pascal.seitz@datadoghq.com>
Bumps [time](https://github.com/time-rs/time) from 0.3.44 to 0.3.47. - [Release notes](https://github.com/time-rs/time/releases) - [Changelog](https://github.com/time-rs/time/blob/main/CHANGELOG.md) - [Commits](time-rs/time@v0.3.44...v0.3.47) --- updated-dependencies: - dependency-name: time dependency-version: 0.3.47 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [bytes](https://github.com/tokio-rs/bytes) from 1.11.0 to 1.11.1. - [Release notes](https://github.com/tokio-rs/bytes/releases) - [Changelog](https://github.com/tokio-rs/bytes/blob/master/CHANGELOG.md) - [Commits](tokio-rs/bytes@v1.11.0...v1.11.1) --- updated-dependencies: - dependency-name: bytes dependency-version: 1.11.1 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
it should warn users against using an incompatible yarn version
* feat(search): add skip_aggregation_finalization to SearchRequest Add optional bool field to SearchRequest that, when true, skips finalization of aggregation results in root.rs and returns the raw IntermediateAggregationResults bytes instead. This enables callers to handle intermediate results directly for multi-step query merging. * test(search): add unit tests for finalize_aggregation_if_any with skip_aggregation_finalization * fix fmt * refactor: change skip_aggregation_finalization from optional bool to bool in proto * fix(test): use tokio::test for SearcherContext which requires Tokio runtime
Add CLAUDE.md with build commands, code conventions, and architecture overview. Include a /fmt skill for format checking and fix the license CSV output path in the Makefile.
The multilang feature pulls heavy dependencies (lindera, whichlang) for little apparent usage. Remove it along with associated code, benchmarks, and tests to reduce build times and binary size.
* feat: Make elastic TermsQuery use TermSetQuery internally Signed-off-by: Darkheir <raphael.cohen@sekoia.io> * Support non-indexed fast fields in TermSetQuery Signed-off-by: Darkheir <raphael.cohen@sekoia.io> --------- Signed-off-by: Darkheir <raphael.cohen@sekoia.io>
* Offload leaf search work to AWS Lambda functions
The goal is to handle traffic spikes gracefully without provisioning additional searcher nodes: when the local search queue is saturated, overflow splits are transparently routed to Lambda for processing.
The offloading decision happens **on the leaf side**, inside the `SearchPermitProvider`. The permit provider already manages a bounded queue of pending split search tasks (gated by memory budget and download slots). When a leaf search request arrives, the provider checks the current queue depth against a configurable `offload_threshold`. If granting permits for all requested splits would exceed this threshold, only enough splits to fill up to the threshold are processed locally — the rest are marked for offloading.
The offloaded splits are batched (up to `max_splits_per_invocation` splits per batch, balanced by document count) and sent to Lambda in parallel. Each Lambda invocation runs the same leaf search code path and **returns per-split results individually**. This is important: the per-split responses are fed back into the `IncrementalCollector` and populate the **partial result cache**, so subsequent queries hitting the same splits benefit from cached results regardless of whether the split was searched locally or on Lambda.
Depending on the configuration, the Lambda function code can be **deployed automatically** at startup. The `quickwit-lambda-client` crate embeds a compressed Lambda binary at compile time. When `auto_deploy` is configured, Quickwit will:
1. Check if a published Lambda version matching the current binary already exists (identified by a description tag `quickwit:{version}-{hash}`)
2. Create or update the function and publish a new version if needed
3. Garbage-collect old versions (keeping the current one + 5 most recent)
This ensures the Lambda function always matches the running Quickwit version without any external deployment tooling. Manual deployment is also supported for users who prefer to manage Lambda functions through Terraform or other IaC tools.
Lambda offloading is opt-in. Add a `lambda` section under `searcher` in the node configuration:
```yaml
searcher:
lambda:
offload_threshold: 100 # queue depth before offloading kicks in (0 = always offload)
max_splits_per_invocation: 10
auto_deploy:
execution_role_arn: arn:aws:iam::123456789012:role/quickwit-lambda-role
memory_size: 5 GiB
invocation_timeout_secs: 15
```
- **`quickwit-lambda-client`**: Handles Lambda invocation (with metrics) and auto-deployment logic. Embeds the Lambda binary at build time.
- **`quickwit-lambda-server`**: The Lambda function handler itself — receives a `LeafSearchRequest`, runs `multi_index_leaf_search`, and returns per-split `LeafSearchResponse`s.
- **`quickwit-search`**: New `LambdaLeafSearchInvoker` trait; `SearchPermitProvider` gains `get_permits_with_offload` to split work between local and offloaded; `leaf.rs` orchestrates local and Lambda tasks in parallel.
- **`quickwit-config`**: New `LambdaConfig` and `LambdaDeployConfig` structs under `SearcherConfig`.
- **`quickwit-serve`**: Initializes the Lambda invoker at startup when configured.
- **`quickwit-proto`**: New `LeafSearchResponses` wrapper message for batched per-split responses.
Bumps [keccak](https://github.com/RustCrypto/sponges) from 0.1.5 to 0.1.6. - [Commits](RustCrypto/sponges@keccak-v0.1.5...keccak-v0.1.6) --- updated-dependencies: - dependency-name: keccak dependency-version: 0.1.6 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Rustfmt 1.9.0 (shipped in nightly 2026-03-01) changed behavior for `imports_granularity = "Module"`: top-level imports are no longer merged. Expand grouped `pub use` re-exports and fix sort order. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* Remove unused React imports from test mocks The automatic JSX transform doesn't require importing React. These unused imports cause biome lint errors. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Update quickwit-ui dependencies Bump minor/patch versions: - @babel/core 7.28.5 → 7.29.0 - @biomejs/biome 2.3.5 → 2.4.4 - @mui/* 7.3.5 → 7.3.8, x-charts/x-date-pickers 8.18.0 → 8.27.x - react/react-dom 19.2.0 → 19.2.4 - react-router 7.12.0 → 7.13.1 - vite 7.2.2 → 7.3.1 - monaco-editor 0.54.0 → 0.55.1 - dayjs 1.11.7 → 1.11.19 - swagger-ui-react 5.30.2 → 5.32.0 - dompurify (resolution) 3.2.4 → 3.3.1 - and other minor bumps Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix stale apt cache in UI CI Cypress step Add apt-get update before installing protobuf-compiler to avoid 404 errors from stale package mirrors on GitHub Actions runners. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Remove unused dependencies react-monaco-editor and web-vitals react-monaco-editor was never imported (project uses @monaco-editor/react). web-vitals and reportWebVitals.ts were dead CRA boilerplate. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Migrate e2e tests from Cypress to Playwright - Replace Cypress with @playwright/test - Create playwright.config.ts with Chromium + baseURL - Translate 3 existing tests + add search page test - Use --only-shell for lighter Chromium install in CI - Exclude e2e/ from Jest via testPathIgnorePatterns Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix biome formatting in homepage.spec.ts Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Refactor UI CI: replace matrix with dedicated jobs Split the matrix-based workflow into two explicit jobs: - `checks`: lint, type check, unit tests (Node-only, no Rust/Postgres) - `e2e`: Playwright tests (with Rust build, Postgres, Chromium) This avoids spinning up Postgres and installing Rust for the lint/type/test jobs that don't need them. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Share Rust cache between CI and UI CI workflows Align RUSTFLAGS in ui-ci.yml with ci.yml and add a shared-key to Swatinem/rust-cache so both workflows reuse the same cached artifacts. This avoids a cold Rust build in the UI e2e job when the main CI has already built on the same branch. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* make lambda support optional * map lambda error to actual variant * retry rate limited lambdas
* WIP * Add `--debug` option to REST API tests runner * Claude WIP * Support Trino ES connector's bool query and range query extensions - Accept `adjust_pure_negative` field in bool queries (emitted by ES Java BoolQueryBuilder, blocks all WHERE predicate pushdown without this fix) - Accept legacy `from`/`to`/`include_lower`/`include_upper` fields in range queries (used by ES Java RangeQueryBuilder instead of standard gt/gte/lt/lte) Co-authored-by: Cursor <cursoragent@cursor.com> * Support wildcard and multi-index patterns in _mappings endpoint The _mapping(s) endpoint now resolves wildcard (`*`) and comma-separated index patterns against the metastore, matching the behavior of real Elasticsearch. This enables Trino's wildcard table feature (e.g. `SELECT * FROM "stack*"`), which calls `GET /stack*/_mappings` to discover schemas across matching indices. Co-authored-by: Cursor <cursoragent@cursor.com> * Extract inline warp filters into filter.rs and revert debug additions Move the inline warp path filters for _aliases and _mappings handlers into dedicated functions in filter.rs, consistent with all other ES compat endpoints. Revert the --debug CLI flag added to run_tests.py as per-step debug: true in YAML files is sufficient. Co-authored-by: Cursor <cursoragent@cursor.com> * Fix clippy collapsible_if and rustfmt formatting Collapse nested if-let + if into if-let chains in range query conversion logic. Fix line length and import ordering for rustfmt. Co-authored-by: Cursor <cursoragent@cursor.com> * Address PR #6168 review comments - bool_query: rename adjust_pure_negative to _adjust_pure_negative with IgnoredAny to convey the field is accepted but unused - bool_query: remove unused let binding in test - range_query: rename json_number to into_json_number, remove doc comments on legacy fields, uncomment from/to conversion logic - cat_indices: add unit test for s parameter validation - mappings: add comprehensive tests for build_properties and merge_dynamic_fields - rest_handler: use advertise_addr for publish_address in _nodes/http and fix es_compat_delete_scroll_handler formatting Made-with: Cursor * bool_query: derive PartialEq instead of manual impl IgnoredAny implements PartialEq (but not Eq), so PartialEq can be derived. Only Eq needs a manual impl since IgnoredAny is a unit struct. Made-with: Cursor * Fix rustfmt nightly formatting for es_compat_delete_scroll_handler Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Adrien Guillo <adrien.guillo@datadoghq.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
(#6191) Co-authored-by: Nicolas Coiffier <n.coiffier@celonis.com>
* feat(obs): export obs as otel * fix: typo * fix: dd license tool
Navigate directly to `/ui` instead of relying on the root redirect, making the tests resilient to changes in the root redirect target. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…6102) * feat(es-compat): add index_filter support for field capabilities API Implements index_filter parameter support for the ES-compatible _field_caps endpoint, allowing users to filter field capabilities based on document queries. Changes: - Add query_ast field to ListFieldsRequest and LeafListFieldsRequest protos - Parse index_filter from ES Query DSL and convert to QueryAst - Pass query_ast through to leaf nodes for future filtering support - Add unit tests for index_filter parsing - Add REST API integration tests Note: This implementation accepts and parses the index_filter parameter for API compatibility. Full split-level document filtering will be added as a follow-up enhancement. Closes #5693 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> Signed-off-by: ruo <ruoliu.dev@gmail.com> * feat(es-compat): implement split-level filtering for field_caps index_filter Address PR review comments for index_filter support in _field_caps API: - Extract `parse_index_filter_to_query_ast()` function with clean prototype - Implement split-level filtering via `split_matches_query()` using lightweight `query.count()` execution (no document materialization) - Add proper async handling with ByteRangeCache, warmup(), and run_cpu_intensive() for Quickwit's async-only storage - Add metastore-level pruning: - Tag extraction via `extract_tags_from_query()` - Time range extraction via `refine_start_end_timestamp_from_ast()` - Build DocMapper only when query_ast is provided (no overhead for common path without index_filter) - Fix REST API tests: use `json:` key (not `json_body:`), use lowercase term values to match tokenizer behavior - Update tests to run against both quickwit and elasticsearch engines Two-level filtering now implemented: 1. Metastore level: tags + time range from query AST 2. Split level: lightweight query execution for accurate filtering Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> Signed-off-by: ruo <ruoliu.dev@gmail.com> * refactor(es-compat): use best-effort metadata filtering for index_filter Remove heavy split-level query execution for field_caps index_filter. The implementation now aligns with ES's "best-effort" approach that uses metadata-level filtering only (time range, tags) instead of opening splits and executing queries. Changes: - Remove split_matches_query function (no longer opens splits) - Remove query_ast and doc_mapper from LeafListFieldsRequest proto - Keep metadata-level filtering in root_list_fields: - Time range extraction from query AST - Tag-based split pruning - Simplify leaf_list_fields to just return fields from all splits This matches ES semantics: "filtering is done on a best-effort basis... this API may return an index even if the provided filter matches no document." Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> Signed-off-by: ruo <ruoliu.dev@gmail.com> * fix(es-compat): reject empty index_filter to match ES behavior - Remove empty object {} handling in parse_index_filter_to_query_ast - ES rejects empty index_filter with 400, now QW does too - Add tag_fields config and tag-based index_filter test - Update unit and integration tests accordingly Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> Signed-off-by: ruo <ruoliu.dev@gmail.com> * Added ref doc for the new functionality. * cargo fmt fix --------- Signed-off-by: ruo <ruoliu.dev@gmail.com> Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> Co-authored-by: fulmicoton <paul.masurel@datadoghq.com>
* Gossip ingester status * Update ingester pool when status changes * Rebalance shards when IngesterStatus changes * Fix timeout_after being 0, causing to not wait for ingester status propagation * Also refresh the ingester pool when an ingester status has changed * Add integration test * Make setup_ingester_pool and setup_indexer_pool a bit more uniform * make fix * Instrument rebalance_shards calls * Unified ClusterNode::for_test_with_ingester_status into ClusterNode::for_test * Remove duplicated readiness check on ingester status * Refactor the ClusterSandbox to add the possibility to dynamically add a node to a running cluster * Ensure the shard is created on the indexer that we shutdown in test_graceful_shutdown_no_data_loss integration test * Don't trigger a rebalance for Add(ready, IngesterStatus::Initializing); improve comments and tests for it * Don't refresh the indexer pool when an indexer update its ingester status * The ingester_pool should contains all ingesters, not only the ready ones * Debug message when no shards to rebalance --------- Co-authored-by: Adrien Guillo <adrien.guillo@datadoghq.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
18 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
When precision is more granular than the second on a timestamp field, we sometimes return events we shouldn't with range requests.
How was this PR tested?
Added an integ test.