From 0ea8a83cbf25ce4202ec16e23e9acc93fcc14d82 Mon Sep 17 00:00:00 2001 From: Peter Barnett Date: Tue, 27 Jan 2026 16:40:16 -0500 Subject: [PATCH 1/6] docs(influxdb3): Add performance upgrade preview documentation for Enterprise Add documentation for the private beta "Performance upgrade preview" feature in InfluxDB 3 Enterprise, which includes major storage engine improvements planned for 3.10 and 3.11 releases. New content: - Overview page with key improvements and enablement instructions - Complete configuration reference for all --pt-* options - Monitoring guide with system tables and troubleshooting Co-Authored-By: Claude Opus 4.5 --- .../enterprise/admin/pachatree/_index.md | 83 ++++ .../enterprise/admin/pachatree/configure.md | 448 ++++++++++++++++++ .../enterprise/admin/pachatree/monitor.md | 316 ++++++++++++ 3 files changed, 847 insertions(+) create mode 100644 content/influxdb3/enterprise/admin/pachatree/_index.md create mode 100644 content/influxdb3/enterprise/admin/pachatree/configure.md create mode 100644 content/influxdb3/enterprise/admin/pachatree/monitor.md diff --git a/content/influxdb3/enterprise/admin/pachatree/_index.md b/content/influxdb3/enterprise/admin/pachatree/_index.md new file mode 100644 index 0000000000..d889454be5 --- /dev/null +++ b/content/influxdb3/enterprise/admin/pachatree/_index.md @@ -0,0 +1,83 @@ +--- +title: Performance upgrade preview +seotitle: Performance upgrade preview for InfluxDB 3 Enterprise +description: > + Preview performance upgrades in InfluxDB 3 Enterprise with improved + single-series query performance, consistent resource usage, wide-and-sparse + table support, and automatic distinct value caching. +menu: + influxdb3_enterprise: + name: Performance upgrade preview + parent: Administer InfluxDB +weight: 115 +influxdb3/enterprise/tags: [storage, performance, beta, preview] +related: + - /influxdb3/enterprise/get-started/setup/ + - /influxdb3/enterprise/admin/pachatree/configure/ + - /influxdb3/enterprise/admin/pachatree/monitor/ + - /influxdb3/enterprise/admin/performance-tuning/ +--- + +> [!Warning] +> #### Private preview beta +> The performance upgrade preview is available to {{% product-name %}} Trial +> and Commercial users as a private beta. These features are subject to breaking changes +> and **should not be used for production workloads**. Your feedback on stability +> and performance at scale helps shape the future of InfluxDB 3. + +{{% product-name %}} includes a private preview of major performance and +feature updates. These updates represent the foundation for the upcoming 3.10 +and 3.11 releases. Available in beta now for InfluxDB 3 Enterprise, many of +these improvements will be coming to InfluxDB 3 Core in subsequent release cycles. + +## Key improvements + +- **Faster single-series queries**: Optimized data path for fetching specific + trends over long time windows, with single-digit millisecond response times. +- **Consistent resource usage**: Reduced CPU and memory spikes during heavy + compaction or ingestion bursts through key-range partitioning. +- **Wide-and-sparse table support**: Handle schemas with up to hundres of thousands of + columns and dynamic schema evolution without expensive rewrites. +- **Automatic distinct value caches**: Dramatically reduced latency for metadata + queries like "show tag values." + +## Enable the preview + +Add the `--use-pacha-tree` flag to your normal +[`influxdb3 serve` startup command](/influxdb3/enterprise/get-started/setup/): + +```bash +influxdb3 serve \ + --node-id host01 \ + --cluster-id cluster01 \ + --object-store file \ + --data-dir ~/.influxdb3 \ + --use-pacha-tree +``` + +The flag exposes additional configuration options prefixed with `--pt-`. +See [Configure the preview](/influxdb3/enterprise/admin/pachatree/configure/) +for tuning options, or [Monitor the preview](/influxdb3/enterprise/admin/pachatree/monitor/) +for system tables and telemetry. + +## Who should try the preview + +Consider enabling the preview in your staging or development environment if +you have workloads with high cardinality, wide tables, frequent backfill, or +query-heavy access patterns. + +Your feedback on stability and speed at scale helps inform the development of +features planned for general availability in the 3.10 and 3.11 releases. + +> [!Note] +> #### Important: Upgraded file format +> These upgrades use an upgraded file format (`.pt` files). When you enable the preview, +> your existing data is automatically upgraded to the new format in place. +> While downgrading is possible, any data written after the upgrade will not +> be included in the downgrade. +> +> Systems with large amounts of data may take time to complete the upgrade. +> For the beta, we recommend using a fresh setup for testing and evaluation +> rather than upgrading existing data. + +{{< children hlevel="h2" readmore=true hr=true >}} diff --git a/content/influxdb3/enterprise/admin/pachatree/configure.md b/content/influxdb3/enterprise/admin/pachatree/configure.md new file mode 100644 index 0000000000..d249711fc9 --- /dev/null +++ b/content/influxdb3/enterprise/admin/pachatree/configure.md @@ -0,0 +1,448 @@ +--- +title: Configure the performance upgrade preview +seotitle: Performance upgrade preview configuration reference for InfluxDB 3 Enterprise +description: > + Complete reference for all configuration options available with the InfluxDB 3 Enterprise + performance upgrades, including memory, compaction, WAL, caching, and replication settings. +menu: + influxdb3_enterprise: + name: Configuration reference + parent: Performance upgrade preview +weight: 202 +influxdb3/enterprise/tags: [storage, configuration, beta, preview, reference] +related: + - /influxdb3/enterprise/admin/pachatree/ + - /influxdb3/enterprise/admin/pachatree/monitor/ + - /influxdb3/enterprise/admin/performance-tuning/ + - /influxdb3/enterprise/reference/config-options/ +--- + +> [!Warning] +> #### Private preview beta +> The performance upgrade preview is available to {{% product-name %}} Trial +> and Commercial users as a private beta. These features are subject to breaking changes +> and **should not be used for production workloads**. Your feedback on stability +> and performance at scale helps shape the future of InfluxDB 3. + +This page provides a complete reference for all configuration options available +with these performance upgrades. All options require the `--use-pacha-tree` flag. + +## Core settings + +| Option | Description | Default | +|:-------|:------------|:--------| +| `--use-pacha-tree` | Enable the performance upgrade preview | `false` | +| `--pt-max-columns` | Maximum total columns across the instance | `6500000` (~6.5M) | + +### Set maximum columns + +Limit the total number of columns across all databases and tables: + +```bash +influxdb3 serve \ + # ... + --use-pacha-tree \ + --pt-max-columns 1000000 +``` + +## Memory management + +Control how the upgraded storage uses memory for buffers, caches, and processing. + +| Option | Description | Default | +|:-------|:------------|:--------| +| `--pt-wal-max-buffer-size` | Maximum WAL buffer size before flush | `15MB` | +| `--pt-snapshot-max-unmerged-size` | Unmerged data threshold for snapshot | `500MB` | +| `--pt-merge-threshold-size` | Size threshold to trigger merge | `125MB` | +| `--pt-replica-max-buffer-size` | Query node buffer maximum | 50% system memory (max 16GB) | + +### WAL buffer size + +The WAL buffer accumulates incoming writes before flushing to object storage. +Larger buffers reduce flush frequency but increase memory usage: + +```bash +influxdb3 serve \ + # ... + --use-pacha-tree \ + --pt-wal-max-buffer-size 30MB +``` + +### Snapshot threshold + +Control when the snapshot buffer triggers a snapshot creation: + +```bash +influxdb3 serve \ + # ... + --use-pacha-tree \ + --pt-snapshot-max-unmerged-size 1GB +``` + +### Merge threshold + +Set the size threshold that triggers background merge operations: + +```bash +influxdb3 serve \ + # ... + --use-pacha-tree \ + --pt-merge-threshold-size 250MB +``` + +### Query node buffer (distributed) + +For query nodes in distributed deployments, control the maximum buffer size: + +```bash +influxdb3 serve \ + # ... + --use-pacha-tree \ + --mode query \ + --pt-replica-max-buffer-size 8GB +``` + +## WAL configuration + +Configure Write-Ahead Log behavior for durability and performance. + +| Option | Description | Default | +|:-------|:------------|:--------| +| `--pt-wal-flush-interval` | Automatic flush interval | `1s` | +| `--pt-wal-merge-on-interval` | Enable interval-based merging | `true` (all-in-one), `false` (multi-node) | +| `--pt-wal-flush-concurrency` | WAL flush worker concurrency | `max(io_threads - 2, 2)` | +| `--pt-wal-replication-interval` | Query node WAL polling interval | `250ms` | + +### Flush interval + +Control how frequently the WAL buffer flushes to object storage: + +```bash +# More frequent flushes (lower latency, more I/O) +influxdb3 serve \ + # ... + --use-pacha-tree \ + --pt-wal-flush-interval 500ms +``` + +```bash +# Less frequent flushes (higher latency, less I/O) +influxdb3 serve \ + # ... + --use-pacha-tree \ + --pt-wal-flush-interval 5s +``` + +### Flush concurrency + +Increase flush workers for high-throughput workloads: + +```bash +influxdb3 serve \ + # ... + --use-pacha-tree \ + --pt-wal-flush-concurrency 8 +``` + +### Interval-based merging + +In all-in-one mode, WAL files are merged on a time interval. +Disable for more control over merge timing: + +```bash +influxdb3 serve \ + # ... + --use-pacha-tree \ + --pt-wal-merge-on-interval false +``` + +## Compaction settings + +Configure background compaction behavior. + +| Option | Description | Default | +|:-------|:------------|:--------| +| `--pt-compactor-concurrency` | Compaction worker concurrency | `max(io_threads - 2, 2)` | +| `--pt-snapshot-merge-concurrency` | Snapshot merge concurrency | `max(io_threads - 2, 2)` | +| `--pt-compactor-cache` | Compaction cache size | 20% of file cache | + +### Compaction concurrency + +Increase compaction workers for faster background processing: + +```bash +influxdb3 serve \ + # ... + --use-pacha-tree \ + --pt-compactor-concurrency 8 +``` + +### Snapshot merge concurrency + +Control parallelism for snapshot merge operations: + +```bash +influxdb3 serve \ + # ... + --use-pacha-tree \ + --pt-snapshot-merge-concurrency 4 +``` + +### Compaction cache + +Dedicated cache for compaction operations (separate from query cache): + +```bash +influxdb3 serve \ + # ... + --use-pacha-tree \ + --pt-compactor-cache 2GB +``` + +## Cache configuration + +Configure data caching for query performance. + +| Option | Description | Default | +|:-------|:------------|:--------| +| `--pt-file-cache-size` | Data file cache size | ~20% system memory | +| `--pt-file-cache-recency` | Only cache files newer than this | `5h` | +| `--pt-file-cache-evict-after` | Evict entries not read within this duration | `24h` | +| `--pt-compactor-cache-ttl` | Compaction cache TTL | `5m` | +| `--pt-disable-data-file-cache` | Disable data file caching | `false` | + +### File cache size + +Set the maximum size for the data file cache: + +```bash +influxdb3 serve \ + # ... + --use-pacha-tree \ + --pt-file-cache-size 8GB +``` + +### Cache recency filter + +Only cache files containing data from the specified time window: + +```bash +influxdb3 serve \ + # ... + --use-pacha-tree \ + --pt-file-cache-recency 24h +``` + +### Cache eviction + +Control how long cached entries persist without access: + +```bash +influxdb3 serve \ + # ... + --use-pacha-tree \ + --pt-file-cache-evict-after 12h +``` + +### Disable caching + +For memory-constrained environments or when using external caching: + +```bash +influxdb3 serve \ + # ... + --use-pacha-tree \ + --pt-disable-data-file-cache +``` + +## File size limits + +Control the size of generated files. + +| Option | Description | Default | +|:-------|:------------|:--------| +| `--pt-gen0-max-rows-per-file` | Maximum rows per Gen0 file | `10000000` (10M) | +| `--pt-gen0-max-bytes-per-file` | Maximum bytes per Gen0 file | `100MB` | + +### Gen0 file size limits + +Control the size of Gen0 files for query and compaction performance: + +```bash +influxdb3 serve \ + # ... + --use-pacha-tree \ + --pt-gen0-max-rows-per-file 5000000 \ + --pt-gen0-max-bytes-per-file 50MB +``` + +## Enterprise replication + +Configure replication behavior for distributed deployments. + +| Option | Description | Default | +|:-------|:------------|:--------| +| `--pt-wal-replica-recovery-concurrency` | Concurrent WAL fetches during recovery | `8` | +| `--pt-wal-replica-steady-concurrency` | Concurrent fetches during steady-state | `8` | +| `--pt-wal-replica-queue-size` | Queue between fetching and merging | `100` | +| `--pt-wal-replica-recovery-tail-skip-limit` | Missing files before stopping recovery | `128` | +| `--pt-replica-gen0-load-concurrency` | Gen0 loads at startup | `16` | +| `--pt-replica-merge-interval` | Query node merge cycle interval | `100ms` | + +### Recovery concurrency + +Control parallelism during query node recovery: + +```bash +influxdb3 serve \ + # ... + --use-pacha-tree \ + --mode query \ + --pt-wal-replica-recovery-concurrency 16 +``` + +### Steady-state replication + +Configure ongoing replication performance: + +```bash +influxdb3 serve \ + # ... + --use-pacha-tree \ + --mode query \ + --pt-wal-replica-steady-concurrency 4 \ + --pt-wal-replica-queue-size 200 +``` + +### Merge interval + +Control how frequently query nodes merge replicated data: + +```bash +influxdb3 serve \ + # ... + --use-pacha-tree \ + --mode query \ + --pt-replica-merge-interval 50ms +``` + +## Other settings + +| Option | Description | Default | +|:-------|:------------|:--------| +| `--pt-enable-retention` | Enforce retention policies | `true` | +| `--pt-disable-hybrid-query` | Disable hybrid Parquet queries | `false` | + +### Retention policies + +Retention policies are enforced by default. Disable for testing or debugging: + +```bash +influxdb3 serve \ + # ... + --use-pacha-tree \ + --pt-enable-retention false +``` + +### Hybrid query mode + +Disable access to legacy Parquet data: + +```bash +influxdb3 serve \ + # ... + --use-pacha-tree \ + --pt-disable-hybrid-query +``` + +## Example configurations + +### Development (minimal resources) + +```bash +influxdb3 serve \ + --node-id dev01 \ + --cluster-id dev \ + --object-store file \ + --data-dir ~/.influxdb3 \ + --use-pacha-tree \ + --pt-file-cache-size 512MB \ + --pt-wal-max-buffer-size 5MB \ + --pt-snapshot-max-unmerged-size 100MB \ + --pt-compactor-concurrency 2 +``` + +### Production all-in-one (8 cores, 32GB RAM) + +```bash +influxdb3 serve \ + --node-id prod01 \ + --cluster-id prod \ + --object-store s3 \ + --bucket YOUR_BUCKET \ + --aws-access-key-id AWS_ACCESS_KEY_ID \ + --aws-secret-access-key AWS_SECRET_ACCESS_KEY \ + --use-pacha-tree \ + --num-io-threads 8 \ + --pt-file-cache-size 8GB \ + --pt-wal-max-buffer-size 30MB \ + --pt-snapshot-max-unmerged-size 1GB \ + --pt-compactor-concurrency 4 \ + --pt-wal-flush-concurrency 4 +``` + +### High-throughput ingest node + +```bash +influxdb3 serve \ + --node-id ingest01 \ + --cluster-id prod \ + --object-store s3 \ + --bucket YOUR_BUCKET \ + --aws-access-key-id AWS_ACCESS_KEY_ID \ + --aws-secret-access-key AWS_SECRET_ACCESS_KEY \ + --use-pacha-tree \ + --mode ingest \ + --num-io-threads 16 \ + --pt-wal-max-buffer-size 50MB \ + --pt-wal-flush-interval 2s \ + --pt-wal-flush-concurrency 8 \ + --pt-snapshot-max-unmerged-size 2GB +``` + +### Query-optimized node + +```bash +influxdb3 serve \ + --node-id query01 \ + --cluster-id prod \ + --object-store s3 \ + --bucket YOUR_BUCKET \ + --aws-access-key-id AWS_ACCESS_KEY_ID \ + --aws-secret-access-key AWS_SECRET_ACCESS_KEY \ + --use-pacha-tree \ + --mode query \ + --num-io-threads 16 \ + --pt-file-cache-size 16GB \ + --pt-file-cache-recency 24h \ + --pt-replica-max-buffer-size 8GB \ + --pt-replica-merge-interval 50ms +``` + +### Dedicated compactor + +```bash +influxdb3 serve \ + --node-id compact01 \ + --cluster-id prod \ + --object-store s3 \ + --bucket YOUR_BUCKET \ + --aws-access-key-id AWS_ACCESS_KEY_ID \ + --aws-secret-access-key AWS_SECRET_ACCESS_KEY \ + --use-pacha-tree \ + --mode compact \ + --num-io-threads 8 \ + --pt-compactor-concurrency 8 \ + --pt-compactor-cache 4GB \ + --pt-compactor-cache-ttl 10m +``` diff --git a/content/influxdb3/enterprise/admin/pachatree/monitor.md b/content/influxdb3/enterprise/admin/pachatree/monitor.md new file mode 100644 index 0000000000..03548fd6ed --- /dev/null +++ b/content/influxdb3/enterprise/admin/pachatree/monitor.md @@ -0,0 +1,316 @@ +--- +title: Monitor the performance upgrade preview +seotitle: Monitor the performance upgrade preview in InfluxDB 3 Enterprise +description: > + Use system tables and query telemetry to monitor file status, query execution, + and overall performance when using InfluxDB 3 Enterprise performance upgrades. +menu: + influxdb3_enterprise: + name: Monitor + parent: Performance upgrade preview +weight: 203 +influxdb3/enterprise/tags: [storage, monitoring, beta, preview, system tables] +related: + - /influxdb3/enterprise/admin/pachatree/ + - /influxdb3/enterprise/admin/pachatree/configure/ + - /influxdb3/enterprise/admin/query-system-data/ +--- + +> [!Warning] +> #### Private preview beta +> The performance upgrade preview is available to {{% product-name %}} Trial +> and Commercial users as a private beta. These features are subject to breaking changes +> and **should not be used for production workloads**. Your feedback on stability +> and performance at scale helps shape the future of InfluxDB 3. + +{{% product-name %}} provides system tables and a query telemetry endpoint to +monitor file status, query execution, and overall performance of these upgrades. + +## System tables + +The upgraded storage exposes internal state through system tables that you can +query with SQL. + +### system.pt_ingest_wal + +View WAL files and their partitions: + +```sql +SELECT * FROM system.pt_ingest_wal; +``` + +Example output: + +| wal_file_id | partition_id | database_id | table_id | min_time | max_time | row_count | size_bytes | +|:------------|:-------------|:------------|:---------|:---------|:---------|:----------|:-----------| +| wal_001 | p_1 | db_1 | t_1 | 2024-01-01T00:00:00Z | 2024-01-01T00:10:00Z | 50000 | 2456789 | +| wal_002 | p_1 | db_1 | t_1 | 2024-01-01T00:10:00Z | 2024-01-01T00:20:00Z | 48000 | 2345678 | + +Use this table to monitor: + +- **WAL accumulation**: Track the number and size of unmerged WAL files +- **Partition distribution**: See how data is distributed across partitions +- **Time coverage**: Verify data time ranges + +#### Monitor WAL backlog + +Check for WAL accumulation that may indicate merging is falling behind: + +```sql +SELECT + COUNT(*) as wal_file_count, + SUM(size_bytes) / 1024 / 1024 as total_size_mb, + MIN(min_time) as oldest_data, + MAX(max_time) as newest_data +FROM system.pt_ingest_wal; +``` + +### system.pt_ingest_files + +View Gen0 files with metadata: + +```sql +SELECT * FROM system.pt_ingest_files; +``` + +Example output: + +| file_id | generation | database_id | table_id | min_time | max_time | row_count | size_bytes | +|:--------|:-----------|:------------|:---------|:---------|:---------|:----------|:-----------| +| gen0_001 | 0 | db_1 | t_1 | 2024-01-01T00:00:00Z | 2024-01-01T01:00:00Z | 500000 | 45678901 | +| gen0_002 | 0 | db_1 | t_1 | 2024-01-01T01:00:00Z | 2024-01-01T02:00:00Z | 480000 | 43567890 | + +Use this table to monitor: + +- **File counts per generation**: Track compaction progress +- **File sizes**: Verify files are within configured limits +- **Time ranges**: Identify time-unbounded Gen0 files + +#### Monitor file distribution + +Check file distribution and compaction status: + +```sql +SELECT + generation, + COUNT(*) as file_count, + SUM(row_count) as total_rows, + SUM(size_bytes) / 1024 / 1024 as total_size_mb, + AVG(size_bytes) / 1024 / 1024 as avg_file_size_mb +FROM system.pt_ingest_files +GROUP BY generation +ORDER BY generation; +``` + +## Query telemetry + +The query telemetry endpoint provides detailed execution statistics for +analyzing query performance. + +### Enable query telemetry + +Query the telemetry endpoint after executing a query: + +```bash +curl -X GET "http://localhost:8181/api/v3/query_sql_telemetry" \ + -H "Authorization: Bearer YOUR_TOKEN" +``` + +### Telemetry response + +The response includes: + +| Field | Description | +|:------|:------------| +| `query_id` | Unique identifier for the query | +| `execution_time_us` | Total execution time in microseconds | +| `chunks` | Per-chunk statistics | +| `cache_stats` | Cache hit rates by type | +| `file_stats` | File-level read statistics | + +### Example telemetry output + +```json +{ + "query_id": "q_12345", + "execution_time_us": 4523, + "chunks": [ + { + "chunk_id": "c_1", + "files_scanned": 3, + "blocks_processed": 12, + "rows_read": 24000, + "rows_returned": 150, + "bytes_read": 1234567 + } + ], + "cache_stats": { + "gen0_hits": 5, + "gen0_misses": 1, + "compacted_hits": 8, + "compacted_misses": 2 + } +} +``` + +## Performance analysis + +### Query performance metrics + +Track these key metrics for query performance: + +| Metric | Good | Warning | Action | +|:-------|:-----|:--------|:-------| +| Cache hit rate | >80% | <60% | Increase cache size or recency window | +| Rows read vs returned ratio | <100:1 | >1000:1 | Add more selective predicates | + +### Ingest performance metrics + +Monitor these metrics for write performance: + +| Metric | Healthy | Warning | Action | +|:-------|:--------|:--------|:-------| +| WAL file count | <50 | >100 | Increase merge concurrency | +| Unmerged size | <500MB | >1GB | Check compaction status | +| Gen0 file count | <100 | >200 | Increase compaction concurrency | + +### Monitor with SQL + +Create a performance summary query: + +```sql +-- File generation summary +SELECT + 'Gen0 files' as metric, + COUNT(*) as count, + SUM(size_bytes) / 1024 / 1024 as size_mb +FROM system.pt_ingest_files +WHERE generation = 0 + +UNION ALL + +SELECT + 'Compacted files' as metric, + COUNT(*) as count, + SUM(size_bytes) / 1024 / 1024 as size_mb +FROM system.pt_ingest_files +WHERE generation > 0 + +UNION ALL + +SELECT + 'WAL files' as metric, + COUNT(*) as count, + SUM(size_bytes) / 1024 / 1024 as size_mb +FROM system.pt_ingest_wal; +``` + +## Troubleshooting + +### High WAL file count + +**Symptom**: `system.pt_ingest_wal` shows many accumulated files. + +**Possible causes**: + +- Merge operations falling behind write rate +- Insufficient merge concurrency +- Object storage latency + +**Solutions**: + +1. Increase merge concurrency: + ```bash + --pt-snapshot-merge-concurrency 8 + ``` + +2. Increase WAL flush interval to create larger, fewer files: + ```bash + --pt-wal-flush-interval 5s + ``` + +3. Check object storage performance and connectivity + +### High cache miss rate + +**Symptom**: `cache_stats` shows >40% miss rate. + +**Possible causes**: + +- Cache size too small for working set +- Cache recency window too narrow +- Random access patterns across time ranges + +**Solutions**: + +1. Increase cache size: + ```bash + --pt-file-cache-size 16GB + ``` + +2. Extend cache recency window: + ```bash + --pt-file-cache-recency 24h + ``` + +3. Extend eviction timeout: + ```bash + --pt-file-cache-evict-after 48h + ``` + +### Slow compaction + +**Symptom**: Gen0 file count continues to grow. + +**Possible causes**: + +- Compaction concurrency too low +- Compaction cache undersized +- High write rate overwhelming compaction + +**Solutions**: + +1. Increase compaction concurrency: + ```bash + --pt-compactor-concurrency 8 + ``` + +2. Increase compaction cache: + ```bash + --pt-compactor-cache 4GB + ``` + +3. For distributed deployments, add dedicated compactor nodes: + ```bash + influxdb3 serve \ + # ... + --use-pacha-tree \ + --mode compact + ``` + +### Query node lag + +**Symptom**: Query nodes return stale data. + +**Possible causes**: + +- Replication falling behind +- Network latency to object storage +- Insufficient replica concurrency + +**Solutions**: + +1. Increase replication concurrency: + ```bash + --pt-wal-replica-steady-concurrency 8 + ``` + +2. Reduce merge interval: + ```bash + --pt-replica-merge-interval 50ms + ``` + +3. Increase replica queue size: + ```bash + --pt-wal-replica-queue-size 200 + ``` From 928913e6eeb39a25951a7ac260e87f3548cd44b7 Mon Sep 17 00:00:00 2001 From: Jason Stirnaman Date: Wed, 28 Jan 2026 11:26:36 -0600 Subject: [PATCH 2/6] Update content/influxdb3/enterprise/admin/pachatree/_index.md Co-authored-by: Scott Anderson --- content/influxdb3/enterprise/admin/pachatree/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/influxdb3/enterprise/admin/pachatree/_index.md b/content/influxdb3/enterprise/admin/pachatree/_index.md index d889454be5..2298c87861 100644 --- a/content/influxdb3/enterprise/admin/pachatree/_index.md +++ b/content/influxdb3/enterprise/admin/pachatree/_index.md @@ -39,7 +39,7 @@ these improvements will be coming to InfluxDB 3 Core in subsequent release cycle - **Wide-and-sparse table support**: Handle schemas with up to hundres of thousands of columns and dynamic schema evolution without expensive rewrites. - **Automatic distinct value caches**: Dramatically reduced latency for metadata - queries like "show tag values." + queries like `SHOW TAG VALUES` in InfluxQL. ## Enable the preview From 8707a37bfc6cec5dbadd298fcb245be0cc4c16e4 Mon Sep 17 00:00:00 2001 From: Jason Stirnaman Date: Wed, 28 Jan 2026 11:26:42 -0600 Subject: [PATCH 3/6] Update content/influxdb3/enterprise/admin/pachatree/_index.md Co-authored-by: Scott Anderson --- content/influxdb3/enterprise/admin/pachatree/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/influxdb3/enterprise/admin/pachatree/_index.md b/content/influxdb3/enterprise/admin/pachatree/_index.md index 2298c87861..4c03d90ac6 100644 --- a/content/influxdb3/enterprise/admin/pachatree/_index.md +++ b/content/influxdb3/enterprise/admin/pachatree/_index.md @@ -69,7 +69,7 @@ query-heavy access patterns. Your feedback on stability and speed at scale helps inform the development of features planned for general availability in the 3.10 and 3.11 releases. -> [!Note] +> [!Important] > #### Important: Upgraded file format > These upgrades use an upgraded file format (`.pt` files). When you enable the preview, > your existing data is automatically upgraded to the new format in place. From 3375d78fcd23aa632ea447e4566d37eac0f19f26 Mon Sep 17 00:00:00 2001 From: Jason Stirnaman Date: Wed, 28 Jan 2026 11:26:49 -0600 Subject: [PATCH 4/6] Update content/influxdb3/enterprise/admin/pachatree/configure.md Co-authored-by: Scott Anderson --- content/influxdb3/enterprise/admin/pachatree/configure.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/influxdb3/enterprise/admin/pachatree/configure.md b/content/influxdb3/enterprise/admin/pachatree/configure.md index d249711fc9..c5b279c5a0 100644 --- a/content/influxdb3/enterprise/admin/pachatree/configure.md +++ b/content/influxdb3/enterprise/admin/pachatree/configure.md @@ -104,7 +104,7 @@ influxdb3 serve \ ## WAL configuration -Configure Write-Ahead Log behavior for durability and performance. +Configure Write-Ahead Log (WAL) behavior for durability and performance. | Option | Description | Default | |:-------|:------------|:--------| From 7c967653467f27a2e0dc5d65dd7c986ad6474bb8 Mon Sep 17 00:00:00 2001 From: Jason Stirnaman Date: Wed, 28 Jan 2026 12:42:29 -0600 Subject: [PATCH 5/6] fix(influxdb3): PR suggestions (#6771) * fix(influxdb3): Content improvements and site notification for performance preview - Fix grammar: 'Handle' -> 'Handles' for consistency - Fix spelling: 'hundres' -> 'hundreds' - Improve file format note wording and change to Important callout - Use AUTH_TOKEN placeholder with proper syntax in monitor.md - Fix unit spacing (500MB -> 500 MB) per style guide - Add site notification for 3.9 performance upgrade preview - Exclude Enterprise pages from 3.8 notification to avoid overlap * Clarify preview feedback instructions and update notification text --- .../enterprise/admin/pachatree/_index.md | 19 +++++++---- .../enterprise/admin/pachatree/configure.md | 10 ++++-- .../enterprise/admin/pachatree/monitor.md | 11 ++++--- data/notifications.yaml | 32 +++++++++++++++++++ 4 files changed, 58 insertions(+), 14 deletions(-) diff --git a/content/influxdb3/enterprise/admin/pachatree/_index.md b/content/influxdb3/enterprise/admin/pachatree/_index.md index 4c03d90ac6..25387ae717 100644 --- a/content/influxdb3/enterprise/admin/pachatree/_index.md +++ b/content/influxdb3/enterprise/admin/pachatree/_index.md @@ -22,7 +22,10 @@ related: > #### Private preview beta > The performance upgrade preview is available to {{% product-name %}} Trial > and Commercial users as a private beta. These features are subject to breaking changes -> and **should not be used for production workloads**. Your feedback on stability +> and **should not be used for production workloads**. +> +> To share feedback on this preview, see [Support and feedback options](#bug-reports-and-feedback). +> Your feedback on stability > and performance at scale helps shape the future of InfluxDB 3. {{% product-name %}} includes a private preview of major performance and @@ -36,8 +39,8 @@ these improvements will be coming to InfluxDB 3 Core in subsequent release cycle trends over long time windows, with single-digit millisecond response times. - **Consistent resource usage**: Reduced CPU and memory spikes during heavy compaction or ingestion bursts through key-range partitioning. -- **Wide-and-sparse table support**: Handle schemas with up to hundres of thousands of - columns and dynamic schema evolution without expensive rewrites. +- **Wide-and-sparse table support**: Handles schemas with up to hundreds of + thousands of columns and dynamic schema evolution without expensive rewrites. - **Automatic distinct value caches**: Dramatically reduced latency for metadata queries like `SHOW TAG VALUES` in InfluxQL. @@ -71,13 +74,15 @@ features planned for general availability in the 3.10 and 3.11 releases. > [!Important] > #### Important: Upgraded file format -> These upgrades use an upgraded file format (`.pt` files). When you enable the preview, +> +> These upgrades use a new file format (`.pt` files). +> When you enable the preview, > your existing data is automatically upgraded to the new format in place. > While downgrading is possible, any data written after the upgrade will not > be included in the downgrade. > -> Systems with large amounts of data may take time to complete the upgrade. -> For the beta, we recommend using a fresh setup for testing and evaluation -> rather than upgrading existing data. +> Systems with large amounts of data may take longer to complete the +> conversion. For the beta, we recommend starting with a fresh setup for +> testing and evaluation rather than converting existing data. {{< children hlevel="h2" readmore=true hr=true >}} diff --git a/content/influxdb3/enterprise/admin/pachatree/configure.md b/content/influxdb3/enterprise/admin/pachatree/configure.md index c5b279c5a0..a6fb1b5e5f 100644 --- a/content/influxdb3/enterprise/admin/pachatree/configure.md +++ b/content/influxdb3/enterprise/admin/pachatree/configure.md @@ -21,9 +21,13 @@ related: > #### Private preview beta > The performance upgrade preview is available to {{% product-name %}} Trial > and Commercial users as a private beta. These features are subject to breaking changes -> and **should not be used for production workloads**. Your feedback on stability +> and **should not be used for production workloads**. +> +> To share feedback on this preview, see [Support and feedback options](#bug-reports-and-feedback). +> Your feedback on stability > and performance at scale helps shape the future of InfluxDB 3. + This page provides a complete reference for all configuration options available with these performance upgrades. All options require the `--use-pacha-tree` flag. @@ -54,7 +58,7 @@ Control how the upgraded storage uses memory for buffers, caches, and processing | `--pt-wal-max-buffer-size` | Maximum WAL buffer size before flush | `15MB` | | `--pt-snapshot-max-unmerged-size` | Unmerged data threshold for snapshot | `500MB` | | `--pt-merge-threshold-size` | Size threshold to trigger merge | `125MB` | -| `--pt-replica-max-buffer-size` | Query node buffer maximum | 50% system memory (max 16GB) | +| `--pt-replica-max-buffer-size` | Query node buffer maximum | 50% system memory (max 16 GB) | ### WAL buffer size @@ -372,7 +376,7 @@ influxdb3 serve \ --pt-compactor-concurrency 2 ``` -### Production all-in-one (8 cores, 32GB RAM) +### Production all-in-one (8 cores, 32 GB RAM) ```bash influxdb3 serve \ diff --git a/content/influxdb3/enterprise/admin/pachatree/monitor.md b/content/influxdb3/enterprise/admin/pachatree/monitor.md index 03548fd6ed..c8c875838b 100644 --- a/content/influxdb3/enterprise/admin/pachatree/monitor.md +++ b/content/influxdb3/enterprise/admin/pachatree/monitor.md @@ -20,7 +20,10 @@ related: > #### Private preview beta > The performance upgrade preview is available to {{% product-name %}} Trial > and Commercial users as a private beta. These features are subject to breaking changes -> and **should not be used for production workloads**. Your feedback on stability +> and **should not be used for production workloads**. +> +> To share feedback on this preview, see [Support and feedback options](#bug-reports-and-feedback). +> Your feedback on stability > and performance at scale helps shape the future of InfluxDB 3. {{% product-name %}} provides system tables and a query telemetry endpoint to @@ -111,9 +114,9 @@ analyzing query performance. Query the telemetry endpoint after executing a query: -```bash +```bash { placeholders="AUTH_TOKEN" } curl -X GET "http://localhost:8181/api/v3/query_sql_telemetry" \ - -H "Authorization: Bearer YOUR_TOKEN" + -H "Authorization: Bearer AUTH_TOKEN" ``` ### Telemetry response @@ -171,7 +174,7 @@ Monitor these metrics for write performance: | Metric | Healthy | Warning | Action | |:-------|:--------|:--------|:-------| | WAL file count | <50 | >100 | Increase merge concurrency | -| Unmerged size | <500MB | >1GB | Check compaction status | +| Unmerged size | <500 MB | >1 GB | Check compaction status | | Gen0 file count | <100 | >200 | Increase compaction concurrency | ### Monitor with SQL diff --git a/data/notifications.yaml b/data/notifications.yaml index 5c513b0ff5..8668e7fa9c 100644 --- a/data/notifications.yaml +++ b/data/notifications.yaml @@ -40,10 +40,42 @@ # - [The plan for InfluxDB 3.0 Open Source](https://influxdata.com/blog/the-plan-for-influxdb-3-0-open-source) # - [InfluxDB 3.0 benchmarks](https://influxdata.com/blog/influxdb-3-0-is-2.5x-45x-faster-compared-to-influxdb-open-source/) +- id: influxdb3.9-performance-preview + level: note + scope: + - /influxdb3/enterprise/ + exclude: + - /influxdb3/enterprise/admin/pachatree/ + title: 'InfluxDB 3.9: Performance upgrade preview' + slug: | + InfluxDB 3 Enterprise 3.9 includes a private preview of major performance upgrades + with faster single-series queries, wide-and-sparse table support, and more. + + message: | + InfluxDB 3 Enterprise 3.9 includes a private preview of major performance and + feature updates. + + **Key improvements:** + + - Faster single-series queries + - Consistent resource usage + - Wide-and-sparse table support + - Automatic distinct value caches for reduced latency with metadata queries + + _Preview features are subject to breaking changes._ + + For more information, see: + + - [Performance upgrade preview](/influxdb3/enterprise/admin/pachatree/) + - [InfluxDB 3 Enterprise release notes](/influxdb3/enterprise/release-notes/) + - [InfluxData Blog post](https://www.influxdata.com/blog/influxdb-3-9/") + - id: influxdb3.8-explorer-1.6 level: note scope: - / + exclude: + - /influxdb3/enterprise/ title: New in InfluxDB 3.8 slug: | Key enhancements in InfluxDB 3.8 and the InfluxDB 3 Explorer 1.6. From 8db83ef2e7799714f573860d02652e7025cbb0dd Mon Sep 17 00:00:00 2001 From: Jason Stirnaman Date: Wed, 4 Feb 2026 17:30:01 -0600 Subject: [PATCH 6/6] Update code blocks with placeholders and clarify AUTH_TOKEN usage (#6773) --- .../enterprise/admin/pachatree/configure.md | 28 +++++++++---------- .../enterprise/admin/pachatree/monitor.md | 3 ++ 2 files changed, 17 insertions(+), 14 deletions(-) diff --git a/content/influxdb3/enterprise/admin/pachatree/configure.md b/content/influxdb3/enterprise/admin/pachatree/configure.md index a6fb1b5e5f..410da02582 100644 --- a/content/influxdb3/enterprise/admin/pachatree/configure.md +++ b/content/influxdb3/enterprise/admin/pachatree/configure.md @@ -363,10 +363,10 @@ influxdb3 serve \ ### Development (minimal resources) -```bash +```bash { placeholders="dev01|dev-cluster"} influxdb3 serve \ --node-id dev01 \ - --cluster-id dev \ + --cluster-id dev-cluster \ --object-store file \ --data-dir ~/.influxdb3 \ --use-pacha-tree \ @@ -378,12 +378,12 @@ influxdb3 serve \ ### Production all-in-one (8 cores, 32 GB RAM) -```bash +```bash { placeholders="prod01|prod-cluster|S3_BUCKET|AWS_ACCESS_KEY_ID|AWS_SECRET_ACCESS_KEY"} influxdb3 serve \ --node-id prod01 \ - --cluster-id prod \ + --cluster-id prod-cluster \ --object-store s3 \ - --bucket YOUR_BUCKET \ + --bucket S3_BUCKET \ --aws-access-key-id AWS_ACCESS_KEY_ID \ --aws-secret-access-key AWS_SECRET_ACCESS_KEY \ --use-pacha-tree \ @@ -397,12 +397,12 @@ influxdb3 serve \ ### High-throughput ingest node -```bash +```bash { placeholders="ingest01|prod-cluster|S3_BUCKET|AWS_ACCESS_KEY_ID|AWS_SECRET_ACCESS_KEY"} influxdb3 serve \ --node-id ingest01 \ - --cluster-id prod \ + --cluster-id prod-cluster \ --object-store s3 \ - --bucket YOUR_BUCKET \ + --bucket S3_BUCKET \ --aws-access-key-id AWS_ACCESS_KEY_ID \ --aws-secret-access-key AWS_SECRET_ACCESS_KEY \ --use-pacha-tree \ @@ -416,12 +416,12 @@ influxdb3 serve \ ### Query-optimized node -```bash +```bash { placeholders="query01|prod-cluster|S3_BUCKET|AWS_ACCESS_KEY_ID|AWS_SECRET_ACCESS_KEY"} influxdb3 serve \ --node-id query01 \ - --cluster-id prod \ + --cluster-id prod-cluster \ --object-store s3 \ - --bucket YOUR_BUCKET \ + --bucket S3_BUCKET \ --aws-access-key-id AWS_ACCESS_KEY_ID \ --aws-secret-access-key AWS_SECRET_ACCESS_KEY \ --use-pacha-tree \ @@ -435,12 +435,12 @@ influxdb3 serve \ ### Dedicated compactor -```bash +```bash { placeholders="compact01|prod-cluster|S3_BUCKET|AWS_ACCESS_KEY_ID|AWS_SECRET_ACCESS_KEY"} influxdb3 serve \ --node-id compact01 \ - --cluster-id prod \ + --cluster-id prod-cluster \ --object-store s3 \ - --bucket YOUR_BUCKET \ + --bucket S3_BUCKET \ --aws-access-key-id AWS_ACCESS_KEY_ID \ --aws-secret-access-key AWS_SECRET_ACCESS_KEY \ --use-pacha-tree \ diff --git a/content/influxdb3/enterprise/admin/pachatree/monitor.md b/content/influxdb3/enterprise/admin/pachatree/monitor.md index c8c875838b..57cf556b6b 100644 --- a/content/influxdb3/enterprise/admin/pachatree/monitor.md +++ b/content/influxdb3/enterprise/admin/pachatree/monitor.md @@ -119,6 +119,9 @@ curl -X GET "http://localhost:8181/api/v3/query_sql_telemetry" \ -H "Authorization: Bearer AUTH_TOKEN" ``` +Replace {{% code-placeholder-key %}}`AUTH_TOKEN`{{% /code-placeholder-key %}} +with your {{% token-link %}}. + ### Telemetry response The response includes: