Skip to content

feat: use DocsFilter#372

Merged
forshev merged 10 commits into
mainfrom
290-use-docs-filter
Apr 10, 2026
Merged

feat: use DocsFilter#372
forshev merged 10 commits into
mainfrom
290-use-docs-filter

Conversation

@forshev
Copy link
Copy Markdown
Collaborator

@forshev forshev commented Mar 5, 2026

Description

Use DocsFilter struct in search and fetch

Fixes #290


  • I have read and followed all requirements in CONTRIBUTING.md;
  • I used LLM/AI assistance to make this pull request;

If you have used LLM/AI assistance please provide model name and full prompt:

Model: {{model-name}}
Prompt: {{prompt}}

@forshev forshev force-pushed the 290-use-docs-filter branch from 14e5d7f to 9eadbb5 Compare March 6, 2026 10:51
@forshev forshev changed the base branch from 290-docs-filter to 290-docs-filter-read-path March 6, 2026 10:52
@eguguchkin eguguchkin added docs-filter feature New feature or request labels Mar 13, 2026
@eguguchkin eguguchkin requested review from cheb0 and dkharms March 16, 2026 10:58
@eguguchkin eguguchkin added this to the v0.71.0 milestone Mar 16, 2026
@forshev forshev force-pushed the 290-docs-filter-read-path branch from 5cb9698 to 7f1ecc3 Compare March 20, 2026 14:21
@forshev forshev force-pushed the 290-use-docs-filter branch 2 times, most recently from 6651e64 to 6001327 Compare March 23, 2026 11:36
@cheb0
Copy link
Copy Markdown
Collaborator

cheb0 commented Mar 31, 2026

@seqbenchbot up main search-keyword-exact-match-warm

@seqbenchbot
Copy link
Copy Markdown
Collaborator

seqbenchbot commented Mar 31, 2026

Oh-oh, @cheb0 ('>o.o)'>!

Something went wrong and I couldn't process your request.
Please take a closer look at error message:

cannot run several comparison benchmarks concurrently

Comment thread frac/processor/eval_tree.go Outdated
Comment thread frac/sealed_index.go Outdated
Comment thread frac/fraction_test.go Outdated
@cheb0
Copy link
Copy Markdown
Collaborator

cheb0 commented Apr 1, 2026

@seqbenchbot up main search-keyword-exact-match-warm

@seqbenchbot
Copy link
Copy Markdown
Collaborator

seqbenchbot commented Apr 1, 2026

Nice, @cheb0 <(-^,^-)=b!

Your request was successfully served.
Identificator for your ongoing benchmark - 76b796bf.

Here is a list of helpful links:

  • Take a look at Grafana dashboard;
  • Live-tailing logs are also available;

Have a great time!

@cheb0
Copy link
Copy Markdown
Collaborator

cheb0 commented Apr 1, 2026

@seqbenchbot down 76b796bf

@seqbenchbot
Copy link
Copy Markdown
Collaborator

seqbenchbot commented Apr 1, 2026

Nice, @cheb0 <(-^,^-)=b!

The benchmark with identificator 76b796bf was finished.
I've prepared a summary for you. Click on Show summary button to see it:

Show summary
Query Type mean (ms) stddev (ms) p(50) (ms) p(95) (ms) p(99) (ms) iterations
base comp diff base comp diff base comp diff base comp diff base comp diff base comp diff
bulk
warm 67.31 68.69 +2.05% 23.91 26.49 +10.79% 61.00 61.00 0.00% 116.50 124.00 +6.44% 151.50 157.50 +3.96% 2450.00 2450.00 0.00%
service:payment-backend-eu
AND k8s_namespace:prod
warm 120.96 122.50 +1.28% 112.08 111.98 -0.09% 99.00 105.00 +6.06% 232.00 234.00 +0.86% 758.50 748.50 -1.32% 7652.00 7407.00 -3.20%

Have a great time!

@dkharms
Copy link
Copy Markdown
Member

dkharms commented Apr 7, 2026

A few comments on files outside the PR diff:

Fraction Refresh

DocsFilter.RefreshFrac is called only after the active fraction is sealed. It re-runs the entire document filtering process, but at that point only LID remapping is necessary.

Issue with Active fraction

I'd consider this a corner case, but it's worth raising. We can receive out-of-order documents (for example, with timestamps several hours before ingestion time), and such documents become available for search and fetch immediately.

It seems DocsFilter should have a separate goroutine that masks data in the active fraction until it is sealed. Otherwise, when a new active fraction is created, func (df *DocsFilter) processFilter(filter *Filter, fracs fracmanager.List) may outpace the write rate.

Comment thread storeapi/store.go Outdated
Comment thread frac/active_index.go
Comment thread frac/active_index.go
@forshev forshev force-pushed the 290-docs-filter-read-path branch from 7f1ecc3 to 19f4b9d Compare April 8, 2026 10:27
@forshev forshev force-pushed the 290-use-docs-filter branch 2 times, most recently from 61259fc to 4d01bdb Compare April 8, 2026 12:03
@forshev forshev mentioned this pull request Apr 9, 2026
2 tasks
@forshev forshev force-pushed the 290-use-docs-filter branch 2 times, most recently from 15f228c to e2d8181 Compare April 9, 2026 14:24
Comment thread cmd/seq-db/seq-db.go
Comment thread cmd/seq-db/seq-db.go Outdated
Comment thread frac/remote.go Outdated
Comment thread filtermanager/filter_manager.go Outdated
Comment thread filtermanager/filter_manager.go Outdated
Comment thread frac/sealed.go Outdated
Comment thread frac/sealed_index.go Outdated
Comment thread fracmanager/fraction_provider.go Outdated
Comment thread storeapi/store.go Outdated
Comment thread tests/integration_tests/integration_test.go Outdated
@forshev forshev force-pushed the 290-use-docs-filter branch from e2d8181 to 6108327 Compare April 10, 2026 07:36
@forshev forshev force-pushed the 290-use-docs-filter branch 2 times, most recently from 5ca791d to 8be1064 Compare April 10, 2026 09:53
@forshev forshev force-pushed the 290-docs-filter-read-path branch from 19f4b9d to c9f85f5 Compare April 10, 2026 13:01
Base automatically changed from 290-docs-filter-read-path to main April 10, 2026 13:08
@forshev forshev force-pushed the 290-use-docs-filter branch from a5aea03 to 27e61d1 Compare April 10, 2026 13:18
@codecov-commenter
Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 70.65217% with 135 lines in your changes missing coverage. Please review.
✅ Project coverage is 71.44%. Comparing base (c9dd219) to head (27e61d1).

Files with missing lines Patch % Lines
skipmaskmanager/skip_mask_manager.go 64.91% 77 Missing and 23 partials ⚠️
cmd/seq-db/seq-db.go 0.00% 15 Missing ⚠️
frac/active_index.go 90.69% 2 Missing and 2 partials ⚠️
skipmaskmanager/encoding.go 57.14% 2 Missing and 1 partial ⚠️
cache/cache.go 75.00% 1 Missing and 1 partial ⚠️
frac/processor/fetch.go 33.33% 1 Missing and 1 partial ⚠️
frac/processor/search.go 77.77% 1 Missing and 1 partial ⚠️
frac/sealed_index.go 93.75% 1 Missing and 1 partial ⚠️
storeapi/store.go 75.00% 1 Missing and 1 partial ⚠️
skipmaskmanager/iterator_asc.go 0.00% 1 Missing ⚠️
... and 2 more
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #372      +/-   ##
==========================================
+ Coverage   70.11%   71.44%   +1.32%     
==========================================
  Files         219      219              
  Lines       16288    16454     +166     
==========================================
+ Hits        11420    11755     +335     
+ Misses       4030     3827     -203     
- Partials      838      872      +34     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@forshev forshev merged commit 124d460 into main Apr 10, 2026
8 checks passed
@forshev forshev deleted the 290-use-docs-filter branch April 10, 2026 14:19
@forshev forshev modified the milestones: v0.71.0, v0.70.0 Apr 10, 2026
@forshev forshev mentioned this pull request Apr 13, 2026
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs-filter feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make better mechanism to hide documents

6 participants