Skip to content

docs: add async search docs #91

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
forshev merged 1 commit into from
Sep 3, 2025
Merged

docs: add async search docs #91

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/en/01-quickstart.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -186,7 +186,7 @@ services:
seq-ui:
image: ghcr.io/ozontech/seq-ui:latest
volumes:
- ${PWD}/config.seq-ui.yaml:/seq-ui-server/config.yaml
- ${PWD}/config.seq-ui.yaml:/seq-ui/config.yaml
ports:
- "5555:5555" # Default HTTP port
- "5556:5556" # Default gRPC port
Expand Down
278 changes: 273 additions & 5 deletions docs/en/10-public-api.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,13 +21,13 @@ seq-db is compatible with Elasticsearch bulk API
Returns a hardcoded ES response that specifies the Elasticsearch version used. This feature allows the usage of seq-db
as an Elasticsearch output for logstash, filebeat, and other log shippers.

#### Example request:
Example request:

```bash
curl -X GET http://localhost:9002/
```

#### Example successful response:
Example successful response:

```json
{
Expand All @@ -42,7 +42,7 @@ curl -X GET http://localhost:9002/

Receives body, parses it to docs and metas and writes to stores via internal API.

#### Example request:
Example request:

```bash
curl -X POST http://localhost:9002/_bulk -d '
Expand All @@ -69,7 +69,7 @@ curl -X POST http://localhost:9002/_bulk -d '
'
```

#### Example successful response:
Example successful response:

```json
{
Expand Down Expand Up @@ -112,7 +112,7 @@ curl -v -X POST http://localhost:9002/_bulk -d '

We get

```
```text
processing doc: unexpected end of string near `", "request_time": "123`
^
```
Expand Down Expand Up @@ -734,3 +734,271 @@ Example successful response:
}
}
```

## Async search gRPC API

### `/StartAsyncSearch`

Start an async search.

Returns an ID that can later be used to fetch results, cancel, or delete the search.

Similar to [`/ComplexSearch`](#complexsearch), but with additional fields: `retention` and `with_docs`.
`retention` determines how long the data will be available.
`with_docs` determines whether document IDs that match the query are found and saved.
If `with_docs` is true, `with_total` is automatically set to true.

Example request:

```bash
grpcurl -plaintext -d '
{
"retention": "3600s",
"query": {
"from": "2025-07-01T05:20:00Z",
"to": "2025-09-01T05:21:00Z",
"query": "message:error | fields level, message"
},
"hist": {
"interval": "1ms"
},
"aggs": [
{
"group_by": "k8s_pod",
"field": "request_time",
"func": "AGG_FUNC_QUANTILE",
"quantiles": [
0.2,
0.8,
0.95
]
}
],
"with_docs": true
}' localhost:9004 seqproxyapi.v1.SeqProxyApi/StartAsyncSearch
```

Example successful response:

```json
{
"search_id": "c28c97d1-117a-45dc-a0cf-d080f22b2a10"
}
Comment thread
forshev marked this conversation as resolved.
```

### `/FetchAsyncSearchResult`

Fetch async search result.

Returns execution result based on a search parameters from [`/StartAsyncSearch`](#startasyncsearch) request.
A partial response can be returned if the search is not yet complete. The `status` field indicates the current status.
Available statuses: `AsyncSearchStatusInProgress`, `AsyncSearchStatusDone`, `AsyncSearchStatusCanceled`, `AsyncSearchStatusError`.

Example request:

```bash
grpcurl -plaintext -d '
{
"search_id": "c28c97d1-117a-45dc-a0cf-d080f22b2a10",
"size": 2,
"offset": 0,
"order": 0
}' localhost:9004 seqproxyapi.v1.SeqProxyApi/FetchAsyncSearchResult
```
Comment thread
forshev marked this conversation as resolved.

Example successful response:

```json
{
"status": "AsyncSearchStatusDone",
"request": {
"retention": "3600s",
"query": {
"query": "message:peggy | fields level, message",
"from": "2025-08-01T05:20:00Z",
"to": "2025-09-01T05:21:00Z"
},
"hist": {
"interval": "1ms"
},
"aggs": [
{
"group_by": "k8s_pod",
"field": "request_time",
"func": "AGG_FUNC_QUANTILE",
"quantiles": [
0.2,
0.8,
0.95
]
}
],
"with_docs": true
},
"response": {
"total": 12,
"docs": [
{
"id": "c09e878998010000-9102a3cb83f156f2",
"data": {
"message": "some error",
"level": 3
},
"time": "2025-08-08T11:53:43.36Z"
},
{
"id": "c09e878998010000-91026cf9cf3386e3",
"data": {
"message": "error 2",
"level": 3
},
"time": "2025-08-08T11:53:43.36Z"
}
Comment thread
forshev marked this conversation as resolved.
],
"aggs": [
{
"buckets": [
{
"docCount": "6",
"key": "seq-proxy",
"value": 6,
"quantiles": [
6,
8,
9
]
}
]
}
],
"hist": {
"buckets": [
{
"docCount": "1",
"ts": "2024-12-23T18:00:36.357Z"
},
{
"docCount": "4",
"ts": "2024-12-23T18:23:41.349Z"
}
]
}
},
"progress": 1,
"disk_usage": "537",
"started_at": "2025-08-08T11:53:52.542336Z",
"expires_at": "2025-08-08T12:53:52.542336Z"
}
Comment thread
forshev marked this conversation as resolved.
```

> `response.docs` and `response.total` fields will be saved and returned only if `with_docs` is true in the [`/StartAsyncSearch`](#startasyncsearch) request

### `/GetAsyncSearchesList`

Get async searches list.

Returns list of available async searches filtered out based on fields from request.
Empty request `{}` returns all available at the moment searches.

Example request:

```bash
grpcurl -plaintext -d '
{
"status": null,
"size": 2,
"offset": 0,
"ids": ["c28c97d1-117a-45dc-a0cf-d080f22b2a10", "8e68d3d7-8e85-44a9-9b09-8de248a3b414"]
}' localhost:9004 seqproxyapi.v1.SeqProxyApi/GetAsyncSearchesList
```

Example successful response:

```json
{
"searches": [
{
"searchId": "c28c97d1-117a-45dc-a0cf-d080f22b2a10",
"status": "AsyncSearchStatusInProgress",
"request": {
"retention": "3600s",
"query": {
"query": "message:error | fields level, message",
"from": "2025-07-01T05:20:00Z",
"to": "2025-08-01T05:21:00Z"
},
"aggs": [],
"with_docs": true
},
"startedAt": "2025-08-08T11:49:20.707200Z",
"expiresAt": "2025-08-08T12:49:20.707200Z",
"progress": 0.89,
"diskUsage": "282"
},
{
"searchId": "8e68d3d7-8e85-44a9-9b09-8de248a3b414",
"status": "AsyncSearchStatusDone",
"request": {
"retention": "3600s",
"query": {
"query": "message:error",
"from": "2025-07-01T05:20:00Z",
"to": "2025-08-01T05:21:00Z"
},
"aggs": [],
"with_docs": false
},
"startedAt": "2025-08-08T11:48:49.488244Z",
"expiresAt": "2025-08-08T12:48:49.488244Z",
"progress": 1,
"diskUsage": "283"
}
]
}
```

### `/CancelAsyncSearch`

Cancel async search.

Cancels async search. Canceled search's execution will be stopped. Can't cancel finished request.

Example request:

```bash
grpcurl -plaintext -d '
{
"search_id": "c28c97d1-117a-45dc-a0cf-d080f22b2a10"
}' localhost:9004 seqproxyapi.v1.SeqProxyApi/CancelAsyncSearch
```
Comment thread
forshev marked this conversation as resolved.

Example successful response:

```json
{}
```

Empty response is successful.

### `/DeleteAsyncSearch`

Delete async search.

Marks async search as expired, the data will be removed the next maintenance iteration.

Example request:

```bash
grpcurl -plaintext -d '
{
"search_id": "c28c97d1-117a-45dc-a0cf-d080f22b2a10"
}' localhost:9004 seqproxyapi.v1.SeqProxyApi/DeleteAsyncSearch
Comment thread
forshev marked this conversation as resolved.
```

Example successful response:

```json
{}
```

Empty response is successful.
31 changes: 31 additions & 0 deletions docs/en/12-async-search.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
---
id: async-search
---

# Async search

Async searches provide the ability to run search requests in the background.

This ability is especially valuable when executing long-running queries that take longer to complete.
Usually this queries are aggregations and histograms.
Async searches can also be used to search documents using `with_docs` field in the [`StartAsyncSearch`](10-public-api.md#startasyncsearch) request, though primary use case is aggregations.

Read [API docs](10-public-api.md#async-search-grpc-api) for more info about the public API.

Async search data is persisted on disk for the specified `retention` time.
Minimum retention is 5 minutes and maximum is 30 days.
Retention is set by the `retention` field in the [`StartAsyncSearch`](10-public-api.md#startasyncsearch) request.
Data is deleted after the retention period expires.

When data size exceeds the limits, read-only mode is enabled: new async searches are rejected, and unfinished searches are suspended until disk space is freed by deleting older searches by retention.

## Configuration

Configuration parameters are:

* `data_dir` [string] - directory that contains async search data. By default, it is a subdirectory of `config.storage.data_dir`.
* `concurrency` [int] - number of concurrent async search executions.
* `max_total_size` [bytes] - maximum total size of all async searches data per store.
* `max_size_per_request` [bytes] - maximum total size of a single async search data per store.

Configuration parameters are part of `async_search` object in the config file.
2 changes: 1 addition & 1 deletion docs/ru/01-quickstart.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -186,7 +186,7 @@ services:
seq-ui:
image: ghcr.io/ozontech/seq-ui:v0.37.0
volumes:
- ${PWD}/config.seq-ui.yaml:/seq-ui-server/config.yaml
- ${PWD}/config.seq-ui.yaml:/seq-ui/config.yaml
ports:
- "5555:5555" # Default HTTP port
- "5556:5556" # Default gRPC port
Expand Down
Loading
Loading