Skip to content

docs: update incremental benchmarks (3.0.4) #354

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
carlos-alm merged 1 commit into from
Mar 7, 2026
Merged

docs: update incremental benchmarks (3.0.4) #354

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 32 additions & 8 deletions generated/benchmarks/INCREMENTAL-BENCHMARKS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@ Import resolution: native batch vs JS fallback throughput.

| Version | Engine | Files | Full Build | No-op | 1-File | Resolve (native) | Resolve (JS) |
|---------|--------|------:|-----------:|------:|-------:|------------------:|-------------:|
| 3.0.4 | native | 177 | 1.0s ↓41% | 319ms ↑7875% | 338ms ↓2% | 4ms ↑6% | 4ms ~ |
Copy link
Contributor

@greptile-apps greptile-apps bot Mar 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Extreme no-op rebuild regression in native engine

The native no-op rebuild time jumped from 4ms → 319ms (+7875%) in 3.0.4. A no-op rebuild (nothing changed) should be one of the fastest operations — just re-checking hashes — so a ~80× slowdown is a strong signal of a real performance bug in the native engine for 3.0.4.

The WASM engine's no-op remains healthy at 5ms (unchanged), which rules out a test/environment artifact and points squarely at a regression in the native Rust path. This may indicate that hash-based change detection in the native engine is no longer short-circuiting correctly, or that some initialization cost is now always paid regardless of whether any files changed.

This benchmark is auto-generated and simply records the measured result, but the anomaly it captures warrants investigation before 3.0.4 is widely adopted.

Rule Used: CLAUDE.md (source)

| 3.0.4 | wasm | 177 | 2.5s ↑6% | 5ms ~ | 568ms ↑12% | 4ms ↑6% | 4ms ~ |
| 3.0.3 | native | 172 | 1.8s ↓9% | 4ms ↓20% | 346ms ~ | 3ms ↓3% | 4ms ↓7% |
| 3.0.3 | wasm | 172 | 2.4s ↓6% | 5ms ↓17% | 505ms ~ | 3ms ↓3% | 4ms ↓7% |
| 3.0.2 | native | 172 | 1.9s ↓17% | 5ms ↑25% | 349ms ↓62% | 4ms ↓3% | 4ms ↑7% |
Expand All @@ -21,37 +23,59 @@ Import resolution: native batch vs JS fallback throughput.

### Latest results

**Version:** 3.0.3 | **Files:** 172 | **Date:** 2026-03-04
**Version:** 3.0.4 | **Files:** 177 | **Date:** 2026-03-06

#### Native (Rust)

| Metric | Value |
|--------|------:|
| Full build | 1.8s |
| No-op rebuild | 4ms |
| 1-file rebuild | 346ms |
| Full build | 1.0s |
| No-op rebuild | 319ms |
| 1-file rebuild | 338ms |

#### WASM

| Metric | Value |
|--------|------:|
| Full build | 2.4s |
| Full build | 2.5s |
| No-op rebuild | 5ms |
| 1-file rebuild | 505ms |
| 1-file rebuild | 568ms |

#### Import Resolution

| Metric | Value |
|--------|------:|
| Import pairs | 202 |
| Native batch | 3ms |
| Native batch | 4ms |
| JS fallback | 4ms |
| Per-import (native) | 0ms |
| Per-import (JS) | 0ms |
| Speedup ratio | 1.2x |
| Speedup ratio | 1.1x |

<!-- INCREMENTAL_BENCHMARK_DATA
[
{
"version": "3.0.4",
"date": "2026-03-06",
"files": 177,
"wasm": {
"fullBuildMs": 2539,
"noopRebuildMs": 5,
"oneFileRebuildMs": 568
},
"native": {
"fullBuildMs": 1030,
"noopRebuildMs": 319,
"oneFileRebuildMs": 338
},
"resolve": {
"imports": 202,
"nativeBatchMs": 3.6,
"jsFallbackMs": 4,
"perImportNativeMs": 0,
"perImportJsMs": 0
}
},
{
"version": "3.0.3",
"date": "2026-03-04",
Expand Down