🚀 Introducing Icecake.Ingest.Streaming — High-Performance Snowpipe Streaming Ingestion for .NET

[jesinity]

I'm excited to announce the first public release of Icecake.Ingest.Streaming, a modern, high-performance .NET library designed to make Snowflake Snowpipe Streaming ingestion fast, reliable, and developer-friendly.

Snowpipe Streaming (v2 API) is Snowflake’s low-latency ingestion channel for real-time workloads — and this library wraps it in a clean, safe, and highly optimized .NET experience.

✨ Key Features

🔥 High-throughput ingestion (internals)

• Efficient double-buffer design
• Zero-overlap flushes via a single-flight semaphore
• Minimal lock contention
• Reusable buffers to reduce GC pressure

🔄 Smart batching
• Flush automatically on:
• max rows
• max bytes
• flush interval
• Client-supplied or automatic offset tokens
• Optional stateful ingestion support

🧱 Battle-tested concurrency model
• Fully async/await .NET http ingestion (of course!)
• Flushing happens outside locks
• Single flush at a time (safe continuation token handling)
• Background timers for:
• periodic flush
• health monitoring

📡 Adaptive offset polling
• FetchLatestCommittedOffsetAsync dynamically adjusts polling frequency
• Uses Snowflake’s SnowflakeAvgProcessingLatencyMs to avoid hammering the endpoint under load