Add lazy workflow/step discovery via deferredEntries in next#976
Conversation
🦋 Changeset detectedLatest commit: f651f1b The changes in this PR will be included in the next version bump. This PR includes changesets to release 14 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
🧪 E2E Test Results❌ Some tests failed Summary
❌ Failed Tests🌍 Community Worlds (41 failed)turso (41 failed):
Details by Category✅ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
❌ 🌍 Community Worlds
✅ 📋 Other
|
📊 Benchmark Results
workflow with no steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Next.js (Turbopack) | Nitro | Express workflow with 1 step💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Nitro | Next.js (Turbopack) workflow with 10 sequential steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Next.js (Turbopack) | Nitro | Express workflow with 25 sequential steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Next.js (Turbopack) | Nitro | Express workflow with 50 sequential steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Express | Next.js (Turbopack) Promise.all with 10 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Nitro | Next.js (Turbopack) Promise.all with 25 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Next.js (Turbopack) | Nitro Promise.all with 50 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Next.js (Turbopack) | Express Promise.race with 10 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Next.js (Turbopack) | Express | Nitro Promise.race with 25 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Next.js (Turbopack) | Express Promise.race with 50 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Nitro | Next.js (Turbopack) Stream Benchmarks (includes TTFB metrics)workflow with stream💻 Local Development
▲ Production (Vercel)
🔍 Observability: Next.js (Turbopack) | Nitro | Express SummaryFastest Framework by WorldWinner determined by most benchmark wins
Fastest World by FrameworkWinner determined by most benchmark wins
Column Definitions
Worlds:
|
c2f801c to
569c87b
Compare
pranaygp
left a comment
There was a problem hiding this comment.
Review: Add lazy workflow/step discovery via deferredEntries in next (#976)
This PR splits the existing NextBuilder into two strategies:
- Eager builder (
builder-eager.ts): The original pre-discovery approach (watchpack-based file watching) - Deferred builder (
builder-deferred.ts): New lazy discovery via Next.jsdeferredEntriesexperimental API + TCP socket communication between loader and builder
The version gate is 16.2.0-canary.30. Overall the architecture is sound - the socket-based discovery channel is well-authenticated and the deferred build queue properly serializes concurrent invocations. Several observations and questions below.
Architecture Observations
-
Builder refactoring is clean: The eager builder is a direct extract of the original code. The deferred builder introduces a fundamentally different lifecycle where
build()only writes stubs and starts the socket server, and actual compilation happens lazily viaonBeforeDeferredEntries(). -
Socket security is reasonable: localhost-only binding + random auth token + message preamble is adequate for local IPC.
-
Build signature deduplication is smart: Using file size + mtime hash to avoid redundant builds is a good optimization.
Issues & Questions
See inline comments for specifics.
# Conflicts: # packages/builders/src/base-builder.ts
This is a revival of #640 which hooks into a new experimental feature I added in Next.js called
deferredEntriesandonBeforeDeferredEntries(vercel/next.js#88347).That feature allows us to bring back lazy discovery as it avoids the race conditions and hacky promise locking in the loader approach we previously tried.
In a follow-up PR we will also update to no longer use
esbuildfor stepsbundling at all which will fix cases like this #766.The eager discovery approach is still needed for older Next.js versions that don't have the
deferredEntriesfeature.x-ref: #523
x-ref: #469
x-ref: #691