Undocumented breaking change in stream.pipeline?

[aduh95]

• Version: 14.0.0
• Platform: any
• Subsystem: stream, doc

What steps will reproduce the bug?

It seems Node.js 14 introduced a breaking change in the stream.pipeline API, however the docs doesn't reference any change related to Node 14.

Refs: max-mapper/extract-zip#94

TLDR of the above issue is that sometimes the promisified version of pipeline never resolves, causing the program to hang or to exit with unfinished promises.

How often does it reproduce? Is there a required condition?

I have a limited knowledge of the Node.js Stream API, I haven't been able to isolate the bug in the issue I referenced above.

What is the expected behavior?

If there is a breaking change on Node.js 14, it should be documented, at least in the YAML metadata.

Ideally there would be a migration guide to workaround the breaking change.

What do you see instead?

Last documented change references 13.10.0.

Additional information

Probably related to #32158.

CC: @ronag

[himself65]

I think we have another undocumented semver-major change

like #32987

[ronag]

#32987

@himself65 how is that related to streams?

[ronag]

@aduh95 Do you think you could provide a full example on how to reproduce?

[himself65]

@himself65 how is that related to streams?

Absolutely not, I just thought this's maybe an undocumented update issue like that one. Because the commit 1428a92 which author mentioned to only release in V14

[ronag]

FYI, I've moved the conversation to max-mapper/extract-zip#94 for now until I know how to reproduce the issue.

[targos]

I had a similar issue with another zip extracting library. I can try to make a repro.

[ronag]

I had a similar issue with another zip extracting library. I can try to make a repro.

Thanks. I'm struggling to make it fail. So far all my tries succeed.

[ronag]

Ok, I managed to make it fail, pre #32967.

@aduh95 Can you confirm whether this is still a problem after #32967?

[targos]

@ronag see https://github.com/targos/bug-zip-pipeline

Expected: node test.js should print two lines (start/finished reading zip file) and the zip file should be fully extracted at result/.

Actual:

• Node.js 13: works
• Node.js 14.0.0: only one file is extracted and one line is printed
• Latest 15.0.0 nightly: only one file is extracted and one line is printed

Edit:

If pipeline is replaced with manual pipe + on('finish'), it works in all versions.

[aduh95]

@aduh95 Can you confirm whether this is still a problem after #32967?

Yes, I can confirm it is still a problem. I am also able to reproduce using @targos repo:

$ npm install
$ ../node/out/Release/node --version
v15.0.0-pre
$ ../node/out/Release/node test.js
start reading zip file
$ node13 --version
v13.13.0
$ node13 test.js 
(node:53994) ExperimentalWarning: The ESM module loader is experimental.
start reading zip file
finished reading zip file

[ronag]

The problem seems to be that the read stream never emits a 'close' event event though it probably should. Either need to fix the read stream or make finished even more strict in regards to when to assume it should wait for 'close'. Digging...

[ronag]

So the problem goes back to fd-slicer which implements its own destroy() function overriding the stream default autoDestroy behavior, thus it never emits 'close'.

https://github.com/andrewrk/node-fd-slicer/blob/master/index.js#L102

At the moment I'm a little unsure how to fix this without basically to large degree disabling the willEmitClose behavior https://github.com/nodejs/node/blob/master/lib/internal/streams/end-of-stream.js#L75.

[ronag]

Neither fd-slicer not yauzl seems to be maintained anymore 😞 so fixing them does not feel likely. thejoshwolfe/yauzl#115

[jasnell]

/cc @mafintosh @mcollina