[world-local] Cross-process streaming for local dev via polling, enable e2e tests#1739
Merged
VaguelySerious merged 6 commits intomainfrom Apr 16, 2026
Merged
[world-local] Cross-process streaming for local dev via polling, enable e2e tests#1739VaguelySerious merged 6 commits intomainfrom
VaguelySerious merged 6 commits intomainfrom
Conversation
Signed-off-by: Peter Wielander <mittgfu@gmail.com>
🦋 Changeset detectedLatest commit: 65e58e4 The changes in this PR will be included in the next version bump. This PR includes changesets to release 19 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
Contributor
🧪 E2E Test Results❌ Some tests failed Summary
❌ Failed Tests🌍 Community Worlds (98 failed)mongodb (15 failed):
redis (15 failed):
turso (68 failed):
Details by Category✅ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
❌ 🌍 Community Worlds
✅ 📋 Other
|
Contributor
📊 Benchmark Results
workflow with no steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Next.js (Turbopack) | Express | Nitro workflow with 1 step💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Next.js (Turbopack) | Nitro workflow with 10 sequential steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Nitro | Next.js (Turbopack) workflow with 25 sequential steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Next.js (Turbopack) | Express workflow with 50 sequential steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Express | Next.js (Turbopack) Promise.all with 10 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Next.js (Turbopack) | Nitro | Express Promise.all with 25 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Nitro | Next.js (Turbopack) Promise.all with 50 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Next.js (Turbopack) | Express Promise.race with 10 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Nitro | Next.js (Turbopack) Promise.race with 25 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Next.js (Turbopack) | Nitro Promise.race with 50 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Express | Next.js (Turbopack) workflow with 10 sequential data payload steps (10KB)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Next.js (Turbopack) | Express workflow with 25 sequential data payload steps (10KB)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Next.js (Turbopack) | Express | Nitro workflow with 50 sequential data payload steps (10KB)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Next.js (Turbopack) | Express workflow with 10 concurrent data payload steps (10KB)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Express | Next.js (Turbopack) workflow with 25 concurrent data payload steps (10KB)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Express | Next.js (Turbopack) workflow with 50 concurrent data payload steps (10KB)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Express | Next.js (Turbopack) Stream Benchmarks (includes TTFB metrics)workflow with stream💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Express | Next.js (Turbopack) stream pipeline with 5 transform steps (1MB)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Next.js (Turbopack) | Express 10 parallel streams (1MB each)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Next.js (Turbopack) | Nitro fan-out fan-in 10 streams (1MB each)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Nitro | Next.js (Turbopack) SummaryFastest Framework by WorldWinner determined by most benchmark wins
Fastest World by FrameworkWinner determined by most benchmark wins
Column Definitions
Worlds:
|
… handling, add test
- Add deliveredChunkIds.has() guard in chunkListener to prevent duplicate
delivery when both EventEmitter and polling are active
- Narrow empty catch {} to only ignore filesystem errors, surface unexpected
errors via console.error
- Add unit test for cross-process polling path (bypasses EventEmitter)
Signed-off-by: Peter Wielander <mittgfu@gmail.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
VaguelySerious
force-pushed
the
peter/enable-tests
branch
from
0bc1dda to
270aad2
Compare
TooTallNate
approved these changes
Apr 16, 2026
Member
TooTallNate
left a comment
TooTallNate
left a comment
There was a problem hiding this comment.
Review: Cross-process streaming via filesystem polling
Clean implementation. The approach is correct — the EventEmitter only works in-process, so polling the shared filesystem at 100ms is the right fallback for the cross-process case (test runner ↔ workbench app).
What looks good
-
Deduplication is correct:
deliveredChunkIdsguards both the EventEmitter path and the polling path. Thehas()+add()are synchronous operations in the same tick, so there's no race between the two delivery mechanisms. -
Ordering is correct: Chunk files use ULID names which are lexicographically sorted (monotonic).
listChunkFilesForStreamsorts them, so polling delivers in chronological order. -
Cleanup is thorough:
pollIntervalis cleared in all exit paths —closeListener(EventEmitter close),cancel()(reader cancels), and inside the poll callback itself (EOF detected).isCompleteandpendingCloseboth return before polling is set up, so no leaked intervals. -
isPollingguard: Prevents overlapping polls if a poll callback takes longer than 100ms (slow filesystem). -
Pre-startIndex tracking: Chunks before
startIndexare added todeliveredChunkIdsso polling doesn't re-deliver them. -
Error handling: Filesystem errors (
ENOENT,EACCES) are silently ignored (transient), non-filesystem errors are surfaced viaconsole.error. -
Test coverage: Unit test directly exercises the cross-process path by writing chunk files to disk (bypassing the EventEmitter), and all previously-skipped e2e stream tests are now enabled.
One nit (non-blocking)
See inline comment about controller.enqueue() on a potentially-closed controller inside the polling callback.
| } | ||
| return; | ||
| } | ||
|
|
Member
There was a problem hiding this comment.
Nit: If the closeListener fires and closes the controller while a poll iteration is in-flight (between the deliveredChunkIds.add() and this controller.enqueue()), this will throw a TypeError ("enqueue on a closed controller"). The outer catch would log it via console.error since TypeError doesn't have a code property — slightly noisy but not harmful.
Could be silenced by adding a let streamClosed = false flag set in the close paths, checked here. But for a local dev tool this is fine as-is.
Member
Author
There was a problem hiding this comment.
(AI) Good catch — added a streamClosed flag that's set in all close paths (closeListener, EOF in polling, isComplete, pendingClose) and checked before controller.enqueue() in the polling callback. This prevents the noisy console.error when closeListener races with an in-flight poll iteration.
ghost
mentioned this pull request
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.