[API Proposal]: Add Deflate, ZLib and GZip encoder/decoder APIs

[AraHaan]

Background and motivation

Currently there is no non-stream based apis to zlib compression. As such I feel like an encoder / decoder implementation is needed similar to the Brotli implementation.

The brotli implementation also uses the encoder / decoder internally in the streams, which can help make the implementations of the zlib based streams (GZipStream, DeflateStream, and ZLibStream) better.

A single ZlibEncoder and ZlibDecoder that takes a class of values (ZlibOptions), where ZlibOptions also has subclasses named DeflateOptions, and GZipOptions where only the window bits are different.

This issue partially addresses:

• Standardize pattern for exposing advanced configuration for compression streams #42820
  • prerequisite of this is: Standardize pattern for exposing advanced configuration for compression streams #42820 (comment) (Comment to the above issue)

Implementing this issue should also resolve this one as well:

• Span-based (non-stream) compression APIs #39327

I currently have a baseline implementation locally of this (except for the stream changes that would need to be done), and it should be ready by the time .NET 7 goes into an api freeze (until .NET 8's development starts).

API Proposal ( updated by @iremyux )

namespace System.IO.Compression
{
   public sealed partial class ZLibCompressionOptions
   {
      public ZLibCompressionOptions() { }
      public int CompressionLevel { get { throw null; } set { } }
      public System.IO.Compression.ZLibCompressionStrategy CompressionStrategy { get { throw null; } set { } }
+     public int WindowLog { get { throw null; } set { } }
   }
}

namespace System.IO.Compression
{  
  public sealed partial class DeflateDecoder : System.IDisposable
  {
      public DeflateDecoder() { }
      public System.Buffers.OperationStatus Decompress(System.ReadOnlySpan<byte> source, System.Span<byte> destination, out int bytesConsumed, out int bytesWritten) { throw null; }
      public void Dispose() { }
      public static bool TryDecompress(System.ReadOnlySpan<byte> source, System.Span<byte> destination, out int bytesWritten) { throw null; }
  }
  public sealed partial class DeflateEncoder : System.IDisposable
  {
      public DeflateEncoder() { }
      public DeflateEncoder(int quality) { }
      public DeflateEncoder(int quality, int windowLog) { }
      public DeflateEncoder(System.IO.Compression.ZLibCompressionOptions options) { }
      public System.Buffers.OperationStatus Compress(System.ReadOnlySpan<byte> source, System.Span<byte> destination, out int bytesConsumed, out int bytesWritten, bool isFinalBlock) { throw null; }
      public void Dispose() { }
      public System.Buffers.OperationStatus Flush(System.Span<byte> destination, out int bytesWritten) { throw null; }
      public static long GetMaxCompressedLength(long inputLength) { throw null; }
      public static bool TryCompress(System.ReadOnlySpan<byte> source, System.Span<byte> destination, out int bytesWritten) { throw null; }
      public static bool TryCompress(System.ReadOnlySpan<byte> source, System.Span<byte> destination, out int bytesWritten, int quality) { throw null; }
      public static bool TryCompress(System.ReadOnlySpan<byte> source, System.Span<byte> destination, out int bytesWritten, int quality, int windowLog) { throw null; }
  }
 public sealed partial class GZipDecoder : System.IDisposable
  {
      public GZipDecoder() { }
      public System.Buffers.OperationStatus Decompress(System.ReadOnlySpan<byte> source, System.Span<byte> destination, out int bytesConsumed, out int bytesWritten) { throw null; }
      public void Dispose() { }
      public static bool TryDecompress(System.ReadOnlySpan<byte> source, System.Span<byte> destination, out int bytesWritten) { throw null; }
  }
  public sealed partial class GZipEncoder : System.IDisposable
  {
      public GZipEncoder() { }
      public GZipEncoder(int quality) { }
      public GZipEncoder(int quality, int windowLog) { }
      public GZipEncoder(System.IO.Compression.ZLibCompressionOptions options) { }
      public System.Buffers.OperationStatus Compress(System.ReadOnlySpan<byte> source, System.Span<byte> destination, out int bytesConsumed, out int bytesWritten, bool isFinalBlock) { throw null; }
      public void Dispose() { }
      public System.Buffers.OperationStatus Flush(System.Span<byte> destination, out int bytesWritten) { throw null; }
      public static long GetMaxCompressedLength(long inputLength) { throw null; }
      public static bool TryCompress(System.ReadOnlySpan<byte> source, System.Span<byte> destination, out int bytesWritten) { throw null; }
      public static bool TryCompress(System.ReadOnlySpan<byte> source, System.Span<byte> destination, out int bytesWritten, int quality) { throw null; }
      public static bool TryCompress(System.ReadOnlySpan<byte> source, System.Span<byte> destination, out int bytesWritten, int quality, int windowLog) { throw null; }
  }
 public sealed partial class ZLibDecoder : System.IDisposable
  {
      public ZLibDecoder() { }
      public System.Buffers.OperationStatus Decompress(System.ReadOnlySpan<byte> source, System.Span<byte> destination, out int bytesConsumed, out int bytesWritten) { throw null; }
      public void Dispose() { }
      public static bool TryDecompress(System.ReadOnlySpan<byte> source, System.Span<byte> destination, out int bytesWritten) { throw null; }
  }
  public sealed partial class ZLibEncoder : System.IDisposable
  {
      public ZLibEncoder() { }
      public ZLibEncoder(int quality) { }
      public ZLibEncoder(int quality, int windowLog) { }
      public ZLibEncoder(System.IO.Compression.ZLibCompressionOptions options) { }
      public System.Buffers.OperationStatus Compress(System.ReadOnlySpan<byte> source, System.Span<byte> destination, out int bytesConsumed, out int bytesWritten, bool isFinalBlock) { throw null; }
      public void Dispose() { }
      public System.Buffers.OperationStatus Flush(System.Span<byte> destination, out int bytesWritten) { throw null; }
      public static long GetMaxCompressedLength(long inputLength) { throw null; }
      public static bool TryCompress(System.ReadOnlySpan<byte> source, System.Span<byte> destination, out int bytesWritten) { throw null; }
      public static bool TryCompress(System.ReadOnlySpan<byte> source, System.Span<byte> destination, out int bytesWritten, int quality) { throw null; }
      public static bool TryCompress(System.ReadOnlySpan<byte> source, System.Span<byte> destination, out int bytesWritten, int quality, int windowLog) { throw null; }
  }
}

API Usage ( updated by @iremyux )

// Simple one-shot compression (Deflate)
byte[] compressed = new byte[DeflateEncoder.GetMaxCompressedLength(data.Length)];
bool success = DeflateEncoder.TryCompress(data, compressed, out int bytesWritten);

// With compression level (ZLib format)
ZLibEncoder.TryCompress(data, compressed, out bytesWritten, CompressionLevel.SmallestSize);

// GZip format
GZipEncoder.TryCompress(data, compressed, out bytesWritten, CompressionLevel.Fastest);

// Stateful for streaming/chunked data
using var encoder = new DeflateEncoder(CompressionLevel.Optimal);
encoder.Compress(chunk1, dest, out int consumed, out int written, isFinalBlock: false);
encoder.Compress(chunk2, dest, out consumed, out written, isFinalBlock: true);

// With strategy
using var encoder2 = new ZLibEncoder(CompressionLevel.Fastest, ZLibCompressionStrategy.HuffmanOnly);

// With fine-grained options (compression level 0-9)
var options = new ZLibCompressionOptions { CompressionLevel = 9, CompressionStrategy = ZLibCompressionStrategy.Filtered };
using var encoder3 = new GZipEncoder(options);

// Decompression
using var decoder = new GZipDecoder();
decoder.Decompress(compressed, decompressed, out consumed, out written);

// One-shot decompression
ZLibDecoder.TryDecompress(compressed, decompressed, out bytesWritten);

Alternative Designs ( updated by @iremyux )

1. We considered having a single ZlibEncoder class with a ZlibCompressionFormat enum parameter (Deflate, ZLib, GZip) to select the output format. However, this would be confusing for users: creating a GZip formatted output from something called "ZlibEncoder" is unintuitive. It also doesn't match the existing pattern where DeflateStream, GZipStream, and ZLibStream are separate classes.

Tagging subscribers to this area: @dotnet/area-system-io-compression
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Background and motivation

Currently there is no non-stream based apis to zlib compression. As such I feel like an encoder / decoder implementation is needed similar to the Brotli implementation.

The brotli implementation also uses the encoder / decoder internally in the streams, which can help make the implementations of the zlib based streams (GZipStream, DeflateStream, and ZLibStream) better.

A single ZlibEncoder and ZlibDecoder that takes a class of values (ZlibOptions), where ZlibOptions also has subclasses named DeflateOptions, and GZipOptions where only the window bits are different.

Dependency issues that will need to be addressed before this:
#42820

Implementing this issue should also resolve this one as well:
#39327

I currently have a baseline implementation locally of this (except for the stream changes that would need to be done), and it should be ready by the time .NET 7 goes into an api freeze (until .NET 8's development starts).

API Proposal

    public enum ZlibOperationStatus
    {
        VersionError = -6,
        BufError,
        MemError,
        DataError,
        StreamError,
        ErrorNo,
        Ok,
        Done = Ok,
        StreamEnd,
        NeedDictionary,
        DestinationTooSmall,
        OperationNotCompression,
        OperationNotDecompression,
        Disposed,
    }
    public struct ZlibEncoder : System.IDisposable
    {
        public ZlibEncoder(ZlibOptions options) { }
        public bool IsDisposed { get { throw null; } }
        public void Dispose() { throw null; }
#pragma warning disable CS3001 // Argument type is not CLS-compliant
        public bool TryCompress(System.ReadOnlySpan<byte> source, System.Span<byte> dest, out uint adler32, out uint bytesWritten) { throw null; }
#pragma warning restore CS3001 // Argument type is not CLS-compliant
#pragma warning disable CS3001 // Argument type is not CLS-compliant
        public ZlibOperationStatus Compress(System.ReadOnlySpan<byte> source, System.Span<byte> dest, out uint adler32, out uint bytesWritten) { throw null; }
#pragma warning restore CS3001 // Argument type is not CLS-compliant
    }
    public struct ZlibDecoder : System.IDisposable
    {
        public ZlibDecoder(ZlibOptions options) { }
        public bool IsDisposed { get { throw null; } }
        public void Dispose() { throw null; }
#pragma warning disable CS3001 // Argument type is not CLS-compliant
        public bool TryDecompress(ReadOnlySpan<byte> source, Span<byte> dest, out uint adler32, out uint bytesWritten, out uint avail) { throw null; }
#pragma warning restore CS3001 // Argument type is not CLS-compliant
#pragma warning disable CS3001 // Argument type is not CLS-compliant
        public ZlibOperationStatus Decompress(ReadOnlySpan<byte> source, Span<byte> dest, out uint adler32, out uint bytesWritten, out uint avail) { throw null; }
#pragma warning restore CS3001 // Argument type is not CLS-compliant
    }

API Usage

// allocate source and dest arrays.
using var zlibEncoder = new ZlibEncoder(new ZlibOptions(ZlibCompressionLevel.Level5));
bool result = zlibEncoder.TryCompress(source, dest, out uint adler, out uint bytesWritten);
if (!result)
{
    Console.WriteLine("Compression failed.");
}
else
{
    Console.WriteLine($"Adler-32: {adler}, Bytes Compressed: {bytesWritten}.");
}

// allocate new source array to compare against the original one.
using var zlibDecoder = new ZlibDecoder(new ZlibOptions());
result = zlibDecoder.TryDecompress(dest, decSource, out adler, out bytesWritten, out uint avail);
if (!result)
{
    Console.WriteLine("Decompression failed.");
}
else
{
    Console.WriteLine($"Adler-32: {adler}, Bytes Decompressed: {bytesWritten}, Available: {avail}.");
}

Alternative Designs

I wanted to use the System.Buffers.OperationStatus enum, but I felt it lacked enough members to denote the zlib specific status codes (where they would be used for zlib, deflate, and gzip compression / decompression). Likewise adler32's and the total_out member from zlib are represented as unsigned, I do not know if I should have the encoder / decoder output signed, or leave them unsigned and keepthe CLS-Compliancy issues suppressed from it.

Risks

Minimal

Author:	AraHaan
Assignees:	-
Labels:	api-suggestion, area-System.IO.Compression, untriaged
Milestone:	-

[AraHaan]

I have modified the proposal somewhat to add a few things I missed to ensure that TryDecompress will never throw or get exceptions that it would need to catch.

[carlossanlop]

@AraHaan thank you so much for the detailed proposal.

This proposal would partially address both #42820 and #39327 . I don't think this PR depends on the first one, we can consider that one the "uber issue", because it was attempting to address the same you're addressing, but for all compression stream classes.

I see you added your own ZLib-specific operation status enums. Contrary to BrotliDecoder/BrotliEncoder, which reused System.Buffers.OperationStatus:

runtime/src/libraries/System.Private.CoreLib/src/System/Buffers/OperationStatus.cs

Line 10 in 57bfe47

public enum OperationStatus

Some feedback on the new enum values:

• Ok and Done seem redundant. We should simply keep Done, just like OperationStatus.Done.
• DestinationTooSmall seems fine. It's aligned to OperationStatus.DestinationTooSmall already.
• I'm not sure we need OperationNotCompression, OperationNotDecompression, Disposed, InitError, ErrorNo. Shouldn't those throw instead?
• What does VersionError mean?
• DataError, StreamEnd, StreamError, BufError, MemError, NeedDictionary, could be aligned to OperationStatus.InvalidData.
• I think DataError could be OperationStatus.NeedMoreData.

What's preventing us from just reusing OperationStatus?

[AraHaan]

I'm not sure we need OperationNotCompression, OperationNotDecompression, Disposed, InitError, ErrorNo. Shouldn't those throw instead?

We would need these for TryDecompress / TryCompress where it would be expansive to throw and then "catch" the exception within the Try version.
VersionError is an zlib error code for when the expected zlib version and the runtime zlib version does not match (on the major version I think).

I guess OperationStatus could add an generic Error onto it at the end of it for these above (see edited proposal above).

According to the Manual DataError is for when "inflate detects an error in the zlib compressed data format, which means that either the data is not a zlib stream to begin with, or that the data was corrupted somewhere along the way since it was compressed."

Also in that manual StreamEnd is returned when inflate/deflate is at the end of the zlib/deflate/gzip stream.

https://zlib.net/zlib_how.html is where I get some of the details as well too.

As for BufError, that might fall under NeedMoreData.

Edit: I have also removed ZlibOperationStatus with a minor change to System.Buffers.OperationStatus.

cc: @carlossanlop I think this might just now be ready for review (hopefully).

[AraHaan]

cc: @dotnet/area-system-io-compression I think this is ready for review.

[AraHaan]

Anything else stalling this?

[rhuijben]

I would like to see some way how many bytes of the original source were read to generate the output, to allow the next decompress to continue right there. Currently the implementation only shows the output buffer results, while the input buffer is just as useful.

This is especially so in case some other information (e.g. not compressed) is written right after the zlib data, such as within Git package files.
The current implementation is nice if you have some outer frame handling that handles/hides overreading input for you, but the zlib streaming was designed to do the framing for you.

[AraHaan]

I think that can be accessed with the LastBytesWritten property (I think it gets changed on every pass).

However I do agree with another thing as well, perhaps there should be an TotalBytesWritten as well for the total amount written in the instance of the Encoder/Decoder.

And possibly an Reset function that can act as an way to reset those 2 get only properties for those who want to be able to reuse it to compress/decompress multiple different sets of input data.