[WIP] Add span-based Deflate, ZLib and GZip encoder/decoder APIs by iremyux · Pull Request #123145 · dotnet/runtime

iremyux · 2026-01-13T18:46:21Z

This PR introduces new span-based, streamless compression and decompression APIs for Deflate, ZLib, and GZip formats, matching the existing BrotliEncoder/BrotliDecoder pattern.

New APIs

DeflateEncoder / DeflateDecoder
ZLibEncoder / ZLibDecoder
GZipEncoder / GZipDecoder

These classes provide:

Instance-based API for streaming/chunked compression with Compress(), Decompress(), and Flush()
Static one-shot API via TryCompress() and TryDecompress() for simple scenarios
GetMaxCompressedLength() to calculate buffer sizes

Closes #62113
Closes #39327

src/libraries/System.IO.Compression/src/System/IO/Compression/ZlibEncoderOptions.cs

src/libraries/System.IO.Compression/src/System/IO/Compression/ZlibEncoder.cs

src/libraries/System.IO.Compression/ref/System.IO.Compression.cs

src/libraries/System.IO.Compression/src/System/IO/Compression/ZlibDecoder.cs

src/libraries/System.IO.Compression/src/System/IO/Compression/ZlibEncoder.cs

rzikm · 2026-01-16T10:40:13Z

src/libraries/System.IO.Compression/src/System/IO/Compression/ZlibEncoder.cs

+        /// <returns>One of the enumeration values that describes the status with which the operation finished.</returns>
+        public OperationStatus Flush(Span<byte> destination, out int bytesWritten)
+        {
+            return Compress(ReadOnlySpan<byte>.Empty, destination, out _, out bytesWritten, isFinalBlock: false);


does this force writing output (if available), I think this should lead to FlushCode.SyncFlush to the native API

rzikm · 2026-01-16T10:44:14Z

src/libraries/System.IO.Compression/src/System/IO/Compression/ZlibEncoder.cs

+        /// <param name="source">A read-only span of bytes containing the source data to compress.</param>
+        /// <param name="destination">When this method returns, a span of bytes where the compressed data is stored.</param>
+        /// <param name="bytesWritten">When this method returns, the total number of bytes that were written to <paramref name="destination"/>.</param>
+        /// <param name="compressionLevel">A number representing compression level. -1 is default, 0 is no compression, 1 is best speed, 9 is best compression.</param>


We should be more clear which default we mean.

Suggested change

/// <param name="compressionLevel">A number representing compression level. -1 is default, 0 is no compression, 1 is best speed, 9 is best compression.</param>

/// <param name="compressionLevel">A number representing compression level. -1 means implementation default, 0 is no compression, 1 is best speed, 9 is best compression.</param>

…mpressedLength

AraHaan · 2026-01-22T23:44:38Z

src/libraries/System.IO.Compression/src/System/IO/Compression/DeflateEncoder.cs

+                CompressionLevel.Fastest => ZLibNative.CompressionLevel.BestSpeed,
+                CompressionLevel.NoCompression => ZLibNative.CompressionLevel.NoCompression,
+                CompressionLevel.SmallestSize => ZLibNative.CompressionLevel.BestCompression,
+                _ => throw new ArgumentOutOfRangeException(nameof(compressionLevel)),


This would fail on valid native compression levels not covered by the CompressionLevel enum. Instead I think it should check if the value is is < -1 or > 9 to throw out of range instead.

Also to add on to the above, now those who want compression levels that just happen to == a value in the CompressionLevel enum will now not be able to use those compression levels either. Perhaps a solution to this is to expose a version of the ctor with CompressionLevel and a version with int that gets casted to ZLibNative.CompressionLevel after a range check.

src/libraries/System.IO.Compression/tests/DeflateZLibGZipEncoderDecoderTests.cs

src/libraries/System.IO.Compression/src/System/IO/Compression/GZipEncoder.cs

Copilot

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 12 comments.

src/libraries/System.IO.Compression/tests/DeflateEncoderDecoderTests.cs

src/libraries/System.IO.Compression/tests/GZipEncoderDecoderTests.cs

Copilot · 2026-02-11T10:48:40Z

src/libraries/System.IO.Compression/tests/GZipEncoderDecoderTests.cs

+            byte[] compressed = new byte[GetMaxCompressedLength(input.Length)];
+            using var encoder = new GZipEncoder(CompressionLevel.Optimal);
+            encoder.Compress(input, compressed, out _, out int compressedSize, isFinalBlock: true);


new GZipEncoder(CompressionLevel.Optimal) won’t compile because GZipEncoder has no CompressionLevel overload. Use the int-quality ctor (e.g., DefaultQuality) or add a CompressionLevel overload and mapping logic.

src/libraries/System.IO.Compression/tests/ZlibEncoderDecoderTests.cs

src/libraries/System.IO.Compression/src/System/IO/Compression/DeflateEncoder.cs

src/libraries/System.IO.Compression/tests/DeflateEncoderDecoderTests.cs

src/libraries/System.IO.Compression/tests/GZipEncoderDecoderTests.cs

Copilot · 2026-02-11T10:48:41Z

src/libraries/System.IO.Compression/tests/ZlibEncoderDecoderTests.cs

+            byte[] input = CreateTestData();
+            using var encoder = new ZLibEncoder(CompressionLevel.Optimal, strategy);
+            byte[] destination = new byte[GetMaxCompressedLength(input.Length)];
+


The tests use ZLibEncoder constructors that take CompressionLevel (and CompressionLevel + ZLibCompressionStrategy), but ZLibEncoder only exposes (int quality), (int quality, int windowLog), and (ZLibCompressionOptions). This won’t compile as-is. Either update the tests to use the available int-quality/options-based constructors, or add the missing public constructors.

src/libraries/System.IO.Compression/src/System/IO/Compression/GZipEncoder.cs

Copilot

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 3 comments.

src/libraries/System.IO.Compression/src/System/IO/Compression/DeflateEncoder.cs

Copilot · 2026-02-12T14:07:54Z

src/libraries/System.IO.Compression/src/System/IO/Compression/DeflateEncoder.cs

+                    OperationStatus status = errorCode switch
+                    {
+                        ZLibNative.ErrorCode.Ok => _state.AvailIn == 0
+                            ? OperationStatus.Done
+                            : _state.AvailOut == 0
+                                ? OperationStatus.DestinationTooSmall
+                                : OperationStatus.Done,
+                        ZLibNative.ErrorCode.StreamEnd => OperationStatus.Done,


When isFinalBlock is true, zlib may return Z_OK (not Z_STREAM_END) indicating the caller must provide more output space and call deflate() again with Z_FINISH until Z_STREAM_END. This implementation can return OperationStatus.Done for ErrorCode.Ok as long as all input was consumed, which can cause callers (especially TryCompress) to treat compression as complete even though the stream trailer hasn’t been fully emitted. Consider returning DestinationTooSmall (or otherwise indicating "need more output") until ErrorCode.StreamEnd is observed.

Copilot · 2026-02-12T14:07:54Z

src/native/libs/System.IO.Compression.Native/pal_zlib.c

+
+uint32_t CompressionNative_CompressBound(uint32_t sourceLen)
+{
+    return (uint32_t)compressBound(sourceLen);


CompressionNative_CompressBound returns uint32_t, but compressBound() returns uLong and can exceed 32-bit for larger inputs; the cast here can silently truncate. Either validate that the result fits in 32-bit (and fail predictably) or widen the exported API to return a 64-bit size (uint64_t/size_t) and update the managed interop accordingly.

Suggested change

return (uint32_t)compressBound(sourceLen);

uLong bound = compressBound(sourceLen);

if (bound > UINT32_MAX)

{

return 0;

}

return (uint32_t)bound;

Copilot

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 2 comments.

Copilot · 2026-02-13T17:04:30Z

src/libraries/System.IO.Compression/src/System/IO/Compression/GZipEncoder.cs

+            // GZip has a larger header than raw deflate, so add extra overhead
+            long baseLength = DeflateEncoder.GetMaxCompressedLength(inputLength);
+
+            // GZip adds 18 bytes: 10-byte header + 8-byte trailer (CRC32 + original size)
+            return baseLength + 18;


The GetMaxCompressedLength calculation for GZip may be incorrect. The native compressBound function calculates the worst-case compressed size including overhead for zlib/deflate format, but it does NOT account for GZip format overhead. According to zlib documentation, compressBound accounts for zlib wrapper (which is 6 bytes) but not GZip wrapper (10-byte header + 8-byte trailer = 18 bytes).

However, when using windowBits with +16 (for GZip), the deflate operation itself will add the GZip header and trailer, so compressBound should be called with the understanding that it returns the bound for the format being used. The issue is that compressBound always calculates based on the zlib wrapper overhead, not taking windowBits into account.

For GZip: The header is 10 bytes, trailer is 8 bytes (total 18). For ZLib: header is 2 bytes, trailer is 4 bytes (total 6). For raw deflate: no wrapper (0 bytes). The compressBound function assumes zlib format (6 bytes overhead).

Therefore:

DeflateEncoder (raw format): needs to subtract 6 bytes from compressBound or calculate differently

ZLibEncoder: compressBound is correct as-is

GZipEncoder: needs to add (18 - 6) = 12 bytes, not 18 bytes

This needs verification against actual zlib behavior, but the current implementation likely overestimates for GZip and may be incorrect for raw Deflate.

Suggested change

// GZip has a larger header than raw deflate, so add extra overhead

long baseLength = DeflateEncoder.GetMaxCompressedLength(inputLength);

// GZip adds 18 bytes: 10-byte header + 8-byte trailer (CRC32 + original size)

return baseLength + 18;

// Base deflate bound (includes zlib wrapper overhead when using compressBound semantics).

long baseLength = DeflateEncoder.GetMaxCompressedLength(inputLength);

// GZip wrapper is 18 bytes (10-byte header + 8-byte trailer), while zlib wrapper is 6 bytes.

// Since the base length already accounts for the 6-byte zlib wrapper, add only the extra 12 bytes.

return baseLength + 12;

Copilot · 2026-02-13T17:04:31Z

src/libraries/System.IO.Compression/src/System/IO/Compression/DeflateEncoder.cs

+        {
+            ArgumentOutOfRangeException.ThrowIfNegative(inputLength);
+            ArgumentOutOfRangeException.ThrowIfGreaterThan(inputLength, uint.MaxValue);
+


The GetMaxCompressedLength for DeflateEncoder may be incorrect for raw deflate format. The native compressBound function calculates worst-case size assuming zlib format (with 6 bytes of wrapper overhead: 2-byte header + 4-byte trailer). However, raw deflate format (used when windowBits is negative) has no wrapper overhead.

This means compressBound overestimates the required buffer size for raw deflate by 6 bytes, which while safe, is not accurate. Consider either:

Subtracting 6 from the result when calculating for raw deflate

Adding a comment explaining that this is a conservative overestimate

The overestimate is small and safe (better than underestimating), so this is a low-priority issue, but it should be documented or corrected for API accuracy.

Suggested change

// Interop.ZLib.compressBound returns a conservative upper bound for zlib-wrapped streams.

// DeflateEncoder uses the raw deflate format, which omits the 2-byte header and 4-byte trailer,

// so this value may be up to 6 bytes larger than strictly necessary, but it is always safe.

Copilot

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated 1 comment.

Copilot · 2026-02-16T11:05:35Z

src/libraries/System.IO.Compression/src/System/IO/Compression/DeflateEncoder.cs

+            ArgumentOutOfRangeException.ThrowIfNegative(inputLength);
+            ArgumentOutOfRangeException.ThrowIfGreaterThan(inputLength, uint.MaxValue);
+
+            return (long)Interop.ZLib.compressBound((uint)inputLength);


The GetMaxCompressedLength calculation is incorrect. The compressBound function from zlib returns the maximum size including the zlib wrapper (6 bytes overhead). For raw Deflate format (no wrapper), the correct calculation should subtract 6 bytes from the compressBound result.

The calculation should be: return (long)Interop.ZLib.compressBound((uint)inputLength) - 6;

Alternatively, a new native function that returns the raw deflate bound could be added.

Suggested change

return (long)Interop.ZLib.compressBound((uint)inputLength);

return (long)Interop.ZLib.compressBound((uint)inputLength) - 6;

@copilot, these are upper bounds, wouldn't overestimating be safer?

iremyux and others added 3 commits January 12, 2026 20:46

Add Zlib Encoder and Decoder classes

e402958

Add tests

95ac930

Merge branch 'dotnet:main' into 62113-zlib-encoder-decoder

88acc57

iremyux self-assigned this Jan 13, 2026

iremyux added the area-System.IO.Compression label Jan 13, 2026

AraHaan reviewed Jan 13, 2026

View reviewed changes

src/libraries/System.IO.Compression/src/System/IO/Compression/ZlibEncoderOptions.cs Outdated Show resolved Hide resolved

AraHaan reviewed Jan 13, 2026

View reviewed changes

src/libraries/System.IO.Compression/src/System/IO/Compression/ZlibEncoderOptions.cs Outdated Show resolved Hide resolved

AraHaan reviewed Jan 13, 2026

View reviewed changes

src/libraries/System.IO.Compression/src/System/IO/Compression/ZlibEncoderOptions.cs Outdated Show resolved Hide resolved

AraHaan reviewed Jan 13, 2026

View reviewed changes

src/libraries/System.IO.Compression/src/System/IO/Compression/ZlibEncoder.cs Outdated Show resolved Hide resolved

AraHaan reviewed Jan 13, 2026

View reviewed changes

src/libraries/System.IO.Compression/ref/System.IO.Compression.cs Outdated Show resolved Hide resolved

This was referenced Jan 13, 2026

"We stopped hearing from agent Azure Pipelines 32. Verify the agent machine is running and has a healthy network connection" dotnet/dnceng#1886

Open

Cannot find 'arm64-v8a' device dotnet/dnceng#2284

Open

rzikm self-requested a review January 14, 2026 11:06

iremyux added 2 commits January 15, 2026 11:28

Use CompressionLevel instead of int

6f1ab80

Use CompressionLevel instead of int

8926759

rzikm reviewed Jan 16, 2026

View reviewed changes

iremyux added 2 commits January 19, 2026 10:51

Introduce seperate encoders and decoders

bb2d89c

Update the tests accordingly

a21a800

iremyux changed the title ~~[WIP] Add span-based ZlibEncoder and ZlibDecoder APIs~~ [WIP] Add span-based Deflate, ZLib and GZip encoder/decoder APIs Jan 19, 2026

iremyux added 4 commits January 19, 2026 11:35

Remove ZlibEncoder and decoder with lower L

4b87dcd

Add ZLibEncoder and Decoder with capital L

aabc3a6

Change GetMaxCompressedLengt's inputLength to long

cfb3d7e

"File is required to end with a single newline character"

f031eed

This was referenced Jan 19, 2026

[android][arm64] System.Net.Sockets.Tests.SendTo_SyncForceNonBlocking.Datagram_UDP_ShouldImplicitlyBindLocalEndpoint fails with NetworkUnreachable #120526

Open

browser-wasm linux LibraryTests build failure - NETSDK1022: Duplicate 'Content' items #123237

Closed

Add Interop layer for zlib's compressBound and use it inside GetMaxCo…

1d18eec

…mpressedLength

build-analysis bot mentioned this pull request Jan 20, 2026

Assertion failures in System.Tests.ConvertTests.TryFromBase64Chars on Mono MiniJIT and Interpreter across multiple Linux and macOS platforms #123389

Closed

AraHaan reviewed Jan 22, 2026

View reviewed changes

rzikm reviewed Jan 23, 2026

View reviewed changes

src/libraries/System.IO.Compression/tests/DeflateZLibGZipEncoderDecoderTests.cs Outdated Show resolved Hide resolved

rzikm reviewed Jan 23, 2026

View reviewed changes

src/libraries/System.IO.Compression/src/System/IO/Compression/GZipEncoder.cs Outdated Show resolved Hide resolved

Add WindowLog to ZLibCompressionOptions

ab9ee4a

Copilot AI review requested due to automatic review settings February 11, 2026 10:37

Copilot started reviewing on behalf of iremyux February 11, 2026 10:38 View session

Copilot AI reviewed Feb 11, 2026

View reviewed changes

iremyux added 2 commits February 12, 2026 14:56

Change the parameter name from inputSize to inputLength

cdb1cb4

Add WindowLog to ZLibCompressionOptions

46c5a8f

Copilot AI review requested due to automatic review settings February 12, 2026 13:57

Copilot started reviewing on behalf of iremyux February 12, 2026 13:58 View session

Copilot AI reviewed Feb 12, 2026

View reviewed changes

iremyux added 2 commits February 13, 2026 12:50

VAlidate inputLength

e35f082

Remove EnsureInitialized

fb4b1c8

build-analysis bot mentioned this pull request Feb 13, 2026

Test failure: baseservices/exceptions/stackoverflow/stackoverflowtester/stackoverflowtester.cmd #110173

Open

Fix tests

f82ddc4

Copilot AI review requested due to automatic review settings February 13, 2026 16:54

Copilot started reviewing on behalf of iremyux February 13, 2026 16:55 View session

Copilot AI reviewed Feb 13, 2026

View reviewed changes

iremyux added 2 commits February 13, 2026 18:08

Fix tests

89ba215

Change the parameter name from inputSize to inputLength

6f05f53

This was referenced Feb 14, 2026

slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes." dotnet/dnceng#1883

Open

Sometimes the helix SDK uses GetWorkItemsAsync when workitems aren't done processing. dotnet/dnceng#6011

Open

iremyux added 3 commits February 16, 2026 11:55

Remove unnecessary parameters

c6384e4

Move test to TestBase

f3dd1c4

Remove unused library

91b2890

Copilot AI review requested due to automatic review settings February 16, 2026 10:59

Copilot started reviewing on behalf of iremyux February 16, 2026 11:00 View session

Copilot AI reviewed Feb 16, 2026

View reviewed changes

	/// <param name="compressionLevel">A number representing compression level. -1 is default, 0 is no compression, 1 is best speed, 9 is best compression.</param>
	/// <param name="compressionLevel">A number representing compression level. -1 means implementation default, 0 is no compression, 1 is best speed, 9 is best compression.</param>

-    return (uint32_t)compressBound(sourceLen);
+    uLong bound = compressBound(sourceLen);
+    if (bound > UINT32_MAX)
+    {
+        return 0;
+    }
+    return (uint32_t)bound;

+            // Interop.ZLib.compressBound returns a conservative upper bound for zlib-wrapped streams.
+            // DeflateEncoder uses the raw deflate format, which omits the 2-byte header and 4-byte trailer,
+            // so this value may be up to 6 bytes larger than strictly necessary, but it is always safe.

	return (long)Interop.ZLib.compressBound((uint)inputLength);
	return (long)Interop.ZLib.compressBound((uint)inputLength) - 6;

Conversation

iremyux commented Jan 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rzikm Jan 16, 2026

Choose a reason for hiding this comment

Uh oh!

rzikm Jan 16, 2026

Choose a reason for hiding this comment

Uh oh!

AraHaan Jan 22, 2026

Choose a reason for hiding this comment

Uh oh!

AraHaan Feb 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Feb 11, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Feb 11, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 12, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Feb 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Feb 16, 2026

Choose a reason for hiding this comment

Uh oh!

iremyux Feb 16, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

iremyux commented Jan 13, 2026 •

edited

Loading

AraHaan Feb 4, 2026 •

edited

Loading