Fix: Stream project ZIP download to prevent 502 timeout and memory issues #3780

[Nixxx19]

Refactored the project download logic to use true end-to-end streaming, eliminating 502 Gateway Timeout errors for large projects (100+ files or >50MB) by preventing memory buffering and ensuring immediate response to gateways.

Fixes #3780

Changes

1. Replaced buffered ZIP generation with streaming

• Changed from generateAsync({ type: 'base64' }) to generateNodeStream({ streamFiles: true })
• Eliminated memory buffering - data flows directly from S3 → JSZip → Client
• Memory usage is now constant O(1) instead of O(n) where n = total project size

2. Sequential asset fetching to prevent connection storms

• Replaced Promise.all() with sequential for...of loop
• Prevents opening 50+ parallel S3 connections
• Processes one file at a time to avoid memory spikes

3. Direct S3 streaming using AWS SDK

• Integrated @aws-sdk/client-s3 with GetObjectCommand for native S3 streaming
• Streams flow directly from S3 without intermediate buffering
• Replaced axios arraybuffer with stream responses

4. Gateway timeout prevention

• Headers sent immediately with Transfer-Encoding: chunked
• Added keepalive mechanism (empty buffer writes every 1 second)
• First data bytes reach gateway within 1-2 seconds (previously 30-60s)

5. Runtime safety improvements

• Added validation for missing root folder
• Graceful handling of missing index.html
• Prevention of duplicate external library entries
• Client disconnect detection and cleanup
• Proper stream error handling

6. Code quality

• Removed unused imports
• Fixed ESLint errors (prefer-destructuring, no-await-in-loop)
• Added comprehensive JSDoc comments

Technical Implementation

Before (Buffered):

const base64 = await zip.generateAsync({ type: 'base64' });
const buff = Buffer.from(base64, 'base64');
res.end(buff);

After (Streaming):

const zipStream = zip.generateNodeStream({
  type: 'nodebuffer',
  streamFiles: true,
  compression: 'DEFLATE'
});
zipStream.pipe(res);

Memory Flow:

S3 → Stream Handle → JSZip → Response Stream → Client
(No intermediate buffering)

---

I have verified that this pull request:

• has no linting errors (npm run lint)
• has no test errors (npm run test)
• has no typecheck errors (npm run typecheck)
• is from a uniquely-named feature branch and is up to date with the develop branch
• is descriptively named and links to an issue number: fix/download-project-streaming-zip-#3780
• meets the standards outlined in the accessibility guidelines

[welcome]

🎉 Thanks for opening this pull request! Please check out our contributing guidelines if you haven't already.

[Nixxx19]

hi @raclim, sorry for the ping!
i wanted to check in on this PR as it addresses a critical issue where large projects fail to download due to gateway timeouts during the zip buffering process. could you please take a look when you have a moment? thanks!

[Nixxx19]

hi @raclim, i've stripped out the redundant comments and stuck to minimal JSdoc for params and returns. changes are pushed. let me know what you think or if you’d like any further refinements.