Memory leak and crash. 1.2.1 regression

[hudochenkov]

• Webpack Version: 4.46.0
• Operating System (or Browser): macOS 10.15.7 (Chrome 87)
• Node Version: 14.15.4
• webpack-plugin-serve Version: 1.2.1

How Do We Reproduce?

To reproduce big repository is needed (where incremental build takes few seconds). I tried to reproduce on recepies in this repo, but couldn't because build step is very fast.

1. Start server.
2. Open browser.
3. Save any file, which triggers webpack build. Half a second later save file again. Second save should happen during compiling time of the first save.

Config:

{
  hmr: false,
  liveReload: true,
  host: '0.0.0.0',
  port: 3000,
  status: false,
  progress: false,
  middleware: (app, builtins) => {
    return builtins.headers({
      'Access-Control-Allow-Origin': '*',
    });
  },
  static: [
    '/Users/alekseyhudochenkov/project/dist'
  ],
  https: {
    key: <Buffer 2d ...>,
    cert: <Buffer 2d ...>
  }
}

Expected Behavior

Build completes. Browser refreshed.

Actual Behavior

CPU usage rises 300+%, and process fails with “JavaScript heap out of memory”.

Stack trace

<--- Last few GCs --->

[63435:0x1046c6000]   100745 ms: Mark-sweep (reduce) 4091.5 (4101.5) -> 4091.1 (4104.3) MB, 2870.1 / 0.0 ms  (average mu = 0.147, current mu = 0.005) allocation failure scavenge might not succeed
[63435:0x1046c6000]   105645 ms: Mark-sweep (reduce) 4092.1 (4105.3) -> 4091.6 (4105.8) MB, 4894.0 / 0.0 ms  (average mu = 0.059, current mu = 0.001) allocation failure scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: MarkCompactCollector: young object promotion failed Allocation failed - JavaScript heap out of memory
 1: 0x1012e4da5 node::Abort() (.cold.1) [/usr/local/bin/node]
 2: 0x1000a6239 node::Abort() [/usr/local/bin/node]
 3: 0x1000a639f node::OnFatalError(char const*, char const*) [/usr/local/bin/node]
 4: 0x1001e9057 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
 5: 0x1001e8ff3 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
 6: 0x100397ee5 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/usr/local/bin/node]
 7: 0x1003f2de3 v8::internal::EvacuateNewSpaceVisitor::Visit(v8::internal::HeapObject, int) [/usr/local/bin/node]
 8: 0x1003da54b void v8::internal::LiveObjectVisitor::VisitBlackObjectsNoFail<v8::internal::EvacuateNewSpaceVisitor, v8::internal::MajorNonAtomicMarkingState>(v8::internal::MemoryChunk*, v8::internal::MajorNonAtomicMarkingState*, v8::internal::EvacuateNewSpaceVisitor*, v8::internal::LiveObjectVisitor::IterationMode) [/usr/local/bin/node]
 9: 0x1003da175 v8::internal::FullEvacuator::RawEvacuatePage(v8::internal::MemoryChunk*, long*) [/usr/local/bin/node]
10: 0x1003d9de6 v8::internal::Evacuator::EvacuatePage(v8::internal::MemoryChunk*) [/usr/local/bin/node]
11: 0x1003f76ee v8::internal::PageEvacuationTask::RunInParallel(v8::internal::ItemParallelJob::Task::Runner) [/usr/local/bin/node]
12: 0x1003b2992 v8::internal::ItemParallelJob::Task::RunInternal() [/usr/local/bin/node]
13: 0x1003b2dff v8::internal::ItemParallelJob::Run() [/usr/local/bin/node]
14: 0x1003dbdd9 void v8::internal::MarkCompactCollectorBase::CreateAndExecuteEvacuationTasks<v8::internal::FullEvacuator, v8::internal::MarkCompactCollector>(v8::internal::MarkCompactCollector*, v8::internal::ItemParallelJob*, v8::internal::MigrationObserver*, long) [/usr/local/bin/node]
15: 0x1003dba11 v8::internal::MarkCompactCollector::EvacuatePagesInParallel() [/usr/local/bin/node]
16: 0x1003c7ad7 v8::internal::MarkCompactCollector::Evacuate() [/usr/local/bin/node]
17: 0x1003c5363 v8::internal::MarkCompactCollector::CollectGarbage() [/usr/local/bin/node]
18: 0x1003985ab v8::internal::Heap::MarkCompact() [/usr/local/bin/node]
19: 0x100394b59 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/usr/local/bin/node]
20: 0x100392911 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/local/bin/node]
21: 0x1003a11aa v8::internal::Heap::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/local/bin/node]
22: 0x1003a1231 v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/local/bin/node]
23: 0x10036ebd7 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/usr/local/bin/node]
24: 0x1006ed918 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/usr/local/bin/node]
25: 0x100a7a279 Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_NoBuiltinExit [/usr/local/bin/node]
zsh: abort      npm start

While I was trying to figure out what dependency causing this at some point I got this stack trace:

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x100a02a19]
    1: StubFrame [pc: 0x100a038ed]
Security context: 0x0e48b27c08d1 <JSObject>
    2: stringifySimple(aka stringifySimple) [0xe4884f31189] [/Users/alekseyhudochenkov/project/node_modules/safe-stable-stringify/stable.js:~453] [pc=0x3004c3e99c76](this=0x0e480c5804b1 <undefined>,15,0x0e48b66b7ff9 <Dependency map = 0xe4803e15ab9>,0x0e48244fcc29 <JSArray[688]>...

Going from 1.2.1 to 1.2.0 don't crash build anymore.

Regression most likely was introduced in #200.

I put console.log(data) in prep function:

webpack-plugin-serve/lib/routes.js

Line 15 in 5a5971f

const prep = (data) => stringify(data);

data argument which causes safe-stable-stringify to fail looks like this:

data

{
  action: 'done',
  data: Stats {
    compilation: Compilation {
      _pluginCompat: [SyncBailHook],
      hooks: [Object],
      name: '0',
      compiler: [Compiler],
      resolverFactory: [ResolverFactory],
      inputFileSystem: [CachedInputFileSystem],
      requestShortener: [RequestShortener],
      options: [Object],
      outputOptions: [Object],
      bail: undefined,
      profile: undefined,
      performance: [Object],
      mainTemplate: [MainTemplate],
      chunkTemplate: [ChunkTemplate],
      hotUpdateChunkTemplate: [HotUpdateChunkTemplate],
      runtimeTemplate: [RuntimeTemplate],
      moduleTemplates: [Object],
      semaphore: [Semaphore],
      entries: [Array],
      _preparedEntrypoints: [Array],
      entrypoints: [Map],
      chunks: [Array],
      chunkGroups: [Array],
      namedChunkGroups: [Map],
      namedChunks: [Map],
      modules: [Array],
      _modules: [Map],
      cache: [Object],
      records: [Object],
      additionalChunkAssets: [],
      assets: [Object],
      assetsInfo: [Map],
      errors: [],
      warnings: [],
      children: [],
      logging: [Map],
      dependencyFactories: [Map],
      dependencyTemplates: [Map],
      childrenCounters: {},
      usedChunkIds: Set(0) {},
      usedModuleIds: null,
      fileTimestamps: [Map],
      contextTimestamps: [Map],
      compilationDependencies: [Set],
      _buildingModules: Map(0) {},
      _rebuildingModules: Map(0) {},
      emittedAssets: Set(0) {},
      fullHash: '0af04a41d10bbf4c9d144d2f89fafa08',
      hash: '0af04a41d10bbf4c9d14',
      fileDependencies: [SortableSet [Set]],
      contextDependencies: [SortableSet [Set]],
      missingDependencies: [SortableSet [Set]]
    },
    hash: '0af04a41d10bbf4c9d14',
    startTime: 1611064073169,
    endTime: 1611064073857
  }
}

Process doesn't fail with

const prep = (data) => JSON.stringify(data);

[shellscape]

Without a reproducible case there's little hope of us fixing this for you. With these types of edge cases we're unfortunately going to need your help in tracking down the issue.

[hudochenkov]

I found someone else's app to show the problem. https://github.com/hudochenkov/Mentor/tree/issue-example

To reproduce:

1. Install dependencies and start server:

   cd packages/web
   npm i
   npm start
   

2. Open http://[::]:55555/ in Chrome, and open dev tools. On Network tab check “Disable cache” and select “Slow 3G” preset.

3. In any code editor open any React component. E. g. /packages/web/src/components/Header.tsx. Press “Save” and roughly half a second later press it again.

4. You should see that node process CPU start to rise and not go down for more than a minute. After that it will crash.

I recorded a video of this, including crash. On the video you'll will see I press an arrow key between two CMD+S. It's for you to see that there is two saves.

Timing could be different on different computers. It's all about finding this spot, when compilation started, but then second compilation triggered. If press Save too quickly it usually doesn't fail. Probably the first compilation cancels, because it's very little time passed. All these jumps through hoops with dev tools is just a way I found to show reproducible example, because I can't share my codebase. In real life situation, when second compilation starts before the first finished happens quite a lot for me in my codebase. Because I save a lot.

When I change prep function mentioned in the first message to use JSON.stringify issue is not reproducible. I think it's all about content of 'done' event, which is stringified (see data summary at the end of the first message). There is non-stringifiable entities or loop.

[shellscape]

If this boils down to safe-stable-stringify it may be worthwhile to open an issue with that project. We're using that package to get around some very real concerns with JSON.stringify, not the least of which are circular references. That's really the best package out there for handling safe stringifying and if they've got a beefy memory leak, they'll want to know about it.

[matheus1lva]

More specifically here: https://github.com/BridgeAR/safe-stable-stringify/blob/master/stable.js#L453

[TrySound]

We use this for logging in our app and didn't see any memory leaks.
https://github.com/moll/json-stringify-safe

[hudochenkov]

Does done needs all this compilation information? Maybe we could keep only needed information and then there might be no issue with stringifying this information.

[shellscape]

Yes, it's important information if doing any kind of diagnostics on incremental builds.