diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md index b6c9d79..1b41e98 100644 --- a/.github/CONTRIBUTING.md +++ b/.github/CONTRIBUTING.md @@ -5,6 +5,75 @@ In the spirit of Open Source Software, everyone is very welcome to contribute to Before contributing to the project, make sure to have a look at our [Code of Conduct](/.github/CODE_OF_CONDUCT.md). +## Development Setup + +### Prerequisites + +- Node.js >= 24 +- npm (comes with Node.js) + +### Getting Started + +```bash +git clone https://github.com/willfarrell/datastream.git +cd datastream +npm install +``` + +This installs all dependencies across all workspaces (`packages/*`, `websites/*`, `.github`). + + +## Project Structure + +``` +packages/ # npm packages (core, csv, compress, encrypt, etc.) +websites/ # documentation website (datastream.js.org) +.github/ # CI workflows, workflow-specific dependencies +bin/ # build scripts +docs/ # additional documentation +``` + +Each package in `packages/` has both a Node.js stream implementation (`index.node.js`) and a Web Streams API implementation (`index.web.js`), selected via conditional exports. + + +## Testing + +```bash +# Run all checks (lint, unit, types, sast, perf, dast) +npm test + +# Run specific test suites +npm run test:lint # Biome linting +npm run test:unit # Unit tests (both Node.js and Web stream variants) +npm run test:unit:node # Unit tests (Node.js streams only) +npm run test:unit:web # Unit tests (Web Streams API only) +npm run test:types # TypeScript type checking (tstyche) +npm run test:perf # Performance benchmarks (tinybench) +npm run test:dast # Fuzz tests (fast-check) + +# Run tests for a single package +node --test ./packages/core +``` + +Unit tests use Node.js built-in `node:test`. Both `--conditions=node` and `--conditions=webstream` are tested to cover both stream implementations. + + +## Building + +```bash +npm run build +``` + +Produces dual ESM builds per package via esbuild: `*.node.mjs` (Node.js) and `*.web.mjs` (Web Streams API), both with external source maps. + + +## Code Style + +- Formatting and linting are handled by [Biome](https://biomejs.dev/) (`biome.json`) +- Commit messages follow [Conventional Commits](https://www.conventionalcommits.org/) enforced by commitlint +- Husky pre-commit hooks run linting and tests automatically + + ## Licence Licensed under [MIT Licence](LICENSE). Copyright (c) 2026 [will Farrell](https://github.com/willfarrell), and the [datastream team](https://github.com/willfarrell/datastream/graphs/contributors). diff --git a/.github/package.json b/.github/package.json index d329f2f..567438f 100644 --- a/.github/package.json +++ b/.github/package.json @@ -1,6 +1,6 @@ { "name": "@datastream/github-workflows", - "version": "0.2.0", + "version": "0.3.1", "private": true, "engines": { "node": ">=24.0" diff --git a/docs/CNAME b/docs/CNAME deleted file mode 100644 index f90c44a..0000000 --- a/docs/CNAME +++ /dev/null @@ -1 +0,0 @@ -datastream.js.org \ No newline at end of file diff --git a/docs/PassThrough.md b/docs/PassThrough.md deleted file mode 100644 index fbd139b..0000000 --- a/docs/PassThrough.md +++ /dev/null @@ -1,12 +0,0 @@ -# Pass Through Stream - -## Streams - -- charset - - detect [passthrough] -- crypto - - digest [passthrough] -- count [passthrough] - - bytes - - chunks -- abort controller \ No newline at end of file diff --git a/docs/Readable.md b/docs/Readable.md deleted file mode 100644 index 96a1cd7..0000000 --- a/docs/Readable.md +++ /dev/null @@ -1,24 +0,0 @@ -# Readable Streams - -- node:fs -- fetch -- ipfs -- aws-s3 - - get [readable] -- pg - - copyfrom [readable] -- postgres - - copyfrom [readable] - - -## Streams - -- charset - - detect [passthrough] - - decode [transform] - - encode [transform] -- crypto - - digest [passthrough] -- csv [transform] - - format - - parse diff --git a/docs/Transform.md b/docs/Transform.md deleted file mode 100644 index fdc2e79..0000000 --- a/docs/Transform.md +++ /dev/null @@ -1,37 +0,0 @@ -# Transform Streams -## Streams - -- charset - - decode [transform] - - encode [transform] -- csv [transform] - - format - - parse -- batch [transform] -- compression [transform] - - brotli - - gzip - - deflate - - protobuf - - zopfli - - zstd - -- json [transform] - - - https://github.com/uhop/stream-json - - https://github.com/creationix/jsonparse - - https://github.com/dominictarr/JSONStream - - https://www.npmjs.com/package/json-stream - - parse - - format - - https://www.npmjs.com/package/json-stream-stringify - - xml [transform] - - parse - - format -- validate [transform] - -- crypto - - encrypt [transform] - - decrypt [transform] -- json-stream - - diff --git a/docs/Writable.md b/docs/Writable.md deleted file mode 100644 index 491e30a..0000000 --- a/docs/Writable.md +++ /dev/null @@ -1,16 +0,0 @@ -# Writable Stream - -## Streams - - -- aws-s3 - - put [writable] -- postgres - - copyto [writable] -- pg - - copyto [writable] -- file -- fetch -- ipfs? - - diff --git a/docs/img/datastream-logo.svg b/docs/img/datastream-logo.svg deleted file mode 100644 index 5fda422..0000000 --- a/docs/img/datastream-logo.svg +++ /dev/null @@ -1 +0,0 @@ - \ No newline at end of file diff --git a/docs/img/logo/Screenshot 2026-02-21 at 19.54.38.png b/docs/img/logo/Screenshot 2026-02-21 at 19.54.38.png deleted file mode 100644 index 2447590..0000000 Binary files a/docs/img/logo/Screenshot 2026-02-21 at 19.54.38.png and /dev/null differ diff --git a/docs/img/logo/Screenshot 2026-02-21 at 20.02.16.png b/docs/img/logo/Screenshot 2026-02-21 at 20.02.16.png deleted file mode 100644 index 8624bfd..0000000 Binary files a/docs/img/logo/Screenshot 2026-02-21 at 20.02.16.png and /dev/null differ diff --git a/docs/packages/_example.md b/docs/packages/_example.md deleted file mode 100644 index 02fddf2..0000000 --- a/docs/packages/_example.md +++ /dev/null @@ -1,72 +0,0 @@ -# @datastream/{package} - -// ToC - -## Setup - -```bash -npm install @datastream/{package} -``` - -## Support - -| Stream | node:stream | node:stream/web | Chrome | Edge | Firefox | Safari | Comments | -| --------------------- | ----------- | --------------- | ------ | ---- | ------- | ------ | ------------ | -| {package}{name}Stream | 18.x | 18.x | 67 | 79 | 102 | 14.1 | Uses ... API | - -### NodeJS - -- pipeline: [15.0.0](https://nodejs.org/dist/latest-v18.x/docs/api/stream.html#streams-promises-api) -- fetch: [18.0.0](https://nodejs.org/en/blog/announcements/v18-release-announce/#fetch-experimental) - -### NodeJS (Web Stream) - -- ReadableStream: [18.0.0](https://nodejs.org/dist/latest-v18.x/docs/api/webstreams.html#new-readablestreamunderlyingsource--strategy) -- TransformStream: [18.0.0](https://nodejs.org/dist/latest-v18.x/docs/api/webstreams.html#new-transformstreamtransformer-writablestrategy-readablestrategy) -- WritableStream: [18.0.0](https://nodejs.org/dist/latest-v18.x/docs/api/webstreams.html#new-writablestreamunderlyingsink-strategy) -- CompressionStream (gzip, deflate): [18.0.0](https://nodejs.org/dist/latest-v18.x/docs/api/webstreams.html#new-compressionstreamformat) -- DecompressionStream (gzip, deflate): [18.0.0](https://nodejs.org/dist/latest-v18.x/docs/api/webstreams.html#new-decompressionstreamformat) -- TextEncoderStream: [18.0.0](https://nodejs.org/dist/latest-v18.x/docs/api/webstreams.html#new-textencoderstream) -- TextDecoderStream: [18.0.0](https://nodejs.org/dist/latest-v18.x/docs/api/webstreams.html#new-textdecoderstreamencoding-options) - -### Browser (Web Stream) - -- ReadableStream: [caniuse](https://caniuse.com/mdn-api_readablestream) - - -- TransformStream: [caniuse](https://caniuse.com/mdn-api_transformstream) -- WritableStream: [caniuse](https://caniuse.com/mdn-api_writablestream) -- CompressionStream (gzip, deflate): [caniuse](https://caniuse.com/mdn-api_compressionstream) -- DecompressionStream (gzip, deflate): [caniuse](https://caniuse.com/mdn-api_decompressionstream) -- TextEncoderStream: [caniuse](https://caniuse.com/mdn-api_textencoderstream) -- TextDecoderStream: [caniuse](https://caniuse.com/mdn-api_textdecoderstream) - -## Help make streams better - -Feature request(s) to +1 - -- W3C: - - webcrypto Streams: https://github.com/w3c/webcrypto/issues/73 - - webcrypto SHA3: https://github.com/w3c/webcrypto/issues/319 -- WHATWG: - - ReadableStream is async iterable: https://github.com/whatwg/streams/issues/778#issuecomment-461341033 -- Web Incubator Community Group (WICG): - - CompressionStream (brotli): https://github.com/WICG/compression/issues/34 - - CompressionStream (zstd): not found -- NodeJS TC39: -- Chrome: -- Firefox: - - CompressionStream: https://bugzilla.mozilla.org/show_bug.cgi?id=1586639 - - TextDecoderStream: https://bugzilla.mozilla.org/show_bug.cgi?id=1486949 - - TextEncoderStream: https://bugzilla.mozilla.org/show_bug.cgi?id=1486949 -- Safari: - - CompressionStream: not found - -## Streams - - - -### {package}{name}Stream ({Readable,PassThrough,Transform,Writable}) - -#### Options - -#### Example(s) diff --git a/docs/packages/aws.md b/docs/packages/aws.md deleted file mode 100644 index 062a233..0000000 --- a/docs/packages/aws.md +++ /dev/null @@ -1,386 +0,0 @@ -# aws - -
-Table of Contents - -
- - -## DynamoDB - - -### awsDynamoDBQueryStream (Readable) - - -Readable stream containing the results of a DynamoDB Query. - -#### Options -See AWS documentation [DynamoDB/Query](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Query.html) - -IAM: `dynamodb:Query` - -#### Egress chunk - -A map of attributes and their values. - -```json -{ - "string" : { - "B": blob, - "BOOL": boolean, - "BS": [ blob ], - "L": [ "AttributeValue" ], - "M": { "string" : "AttributeValue" }, - "N": "string", - "NS": [ "string" ], - "NULL": boolean, - "S": "string", - "SS": [ "string" ] - } -} -``` - -See AWS documentation [DynamoDB/Query](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Query.html) for `Responses` structure. - -#### Example - -```javascript -import { pipeline } from '@datastream/core' -import { awsDynamoDBQueryStream } from '@datastream/aws/dynamodb' - -const streams = [ - await awsDynamoDBQueryStream(options), - ... -] - -await pipeline(streams) -``` - - -### awsDynamoDBScanStream (Readable) - - -Readable stream containing the results of a DynamoDB Scan. - -#### Options -See AWS documentation [DynamoDB/Scan](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Scan.html) - -IAM: `dynamodb:Scan` - -#### Egress chunk - -A map of attributes and their values. - -```json -{ - "string" : { - "B": blob, - "BOOL": boolean, - "BS": [ blob ], - "L": [ "AttributeValue" ], - "M": { "string" : "AttributeValue" }, - "N": "string", - "NS": [ "string" ], - "NULL": boolean, - "S": "string", - "SS": [ "string" ] - } -} -``` - -See AWS documentation [DynamoDB/Scan](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Scan.html) for `Responses` structure. - -#### Example - -```javascript -import { pipeline } from '@datastream/core' -import { awsDynamoDBScanStream } from '@datastream/aws/dynamodb' - -const streams = [ - await awsDynamoDBScanStream(options), - ... -] - -await pipeline(streams) -``` - -### awsDynamoDBGetItemStream (Readable) - - -Readable stream containing the results of a DynamoDB BatchGetItems. - -#### Options - -- `TableName` (string): Name of the table to get items from. -- `Keys` (object[]): See AWS documentation [DynamoDB/BatchGetItem](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_BatchGetItem.html) for `Keys` structure. -- `retryCount` (int) [0]: Starting retry count used for back-off timer, `3 ^ retryCount` -- `retryMaxCount` (int) [10]: Max number of retries before stopping - -IAM: `dynamodb:BatchGetItem` - -#### Egress chunk - -A map of attributes and their values. - -```json -{ - "string" : { - "B": blob, - "BOOL": boolean, - "BS": [ blob ], - "L": [ "AttributeValue" ], - "M": { "string" : "AttributeValue" }, - "N": "string", - "NS": [ "string" ], - "NULL": boolean, - "S": "string", - "SS": [ "string" ] - } -} -``` - -See AWS documentation [DynamoDB/BatchGetItem](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_BatchGetItem.html) for `Responses` structure. - -#### Example - -```javascript -import { pipeline } from '@datastream/core' -import { awsDynamoDBGetItemStream } from '@datastream/aws/dynamodb' - -const streams = [ - await awsDynamoDBGetItemStream(options), - ... -] - -await pipeline(streams) -``` - -### awsDynamoDBPutItemStream (Writable) - - -Writable stream that sends items to DynamoDB BatchWriteItems. - -#### Options - -- `TableName` (string): Name of the table to get items from. -- `retryCount` (int) [0]: Starting retry count used for back-off timer, `3 ^ retryCount` -- `retryMaxCount` (int) [10]: Max number of retries before stopping - -IAM: `dynamodb:BatchWriteItem` - -#### Ingress chunk -A map of attributes and their values. Each entry in this map consists of an attribute name and an attribute value. Attribute values must not be null; string and binary type attributes must have lengths greater than zero; and set type attributes must not be empty. Requests that contain empty values are rejected with a ValidationException exception. - -If you specify any attributes that are part of an index key, then the data types for those attributes must match those of the schema in the table's attribute definition. - -```json -{ - "string" : { - "B": blob, - "BOOL": boolean, - "BS": [ blob ], - "L": [ "AttributeValue" ], - "M": { "string" : "AttributeValue" }, - "N": "string", - "NS": [ "string" ], - "NULL": boolean, - "S": "string", - "SS": [ "string" ] - } -} -``` - -See AWS documentation [DynamoDB/BatchWriteItem.PutRequest.Item](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_BatchWriteItem.html) - -#### Example - -```javascript -import { pipeline } from '@datastream/core' -import { awsDynamoDBPutItemStream } from '@datastream/aws/dynamodb' - -const streams = [ - ... - await awsDynamoDBPutItemStream(options) -] - -await pipeline(streams) -``` - -### awsDynamoDBDeleteItemStream (Writable) - - -Writable stream that sends keys to DynamoDB BatchWriteItems. - -#### Options - -- `TableName` (string) Required: Name of the table to get items from. -- `retryCount` (int) Optional (0): Starting retry count used for back-off timer, `3 ^ retryCount` -- `retryMaxCount` (int) Optional (10): Max number of retries before stopping - -IAM: `dynamodb:BatchWriteItem` - -#### Ingress chunk -A map of primary key attribute values that uniquely identify the item. Each entry in this map consists of an attribute name and an attribute value. For each primary key, you must provide all of the key attributes. For example, with a simple primary key, you only need to provide a value for the partition key. For a composite primary key, you must provide values for both the partition key and the sort key. - - -```json -{ - "string" : { - "B": blob, - "BOOL": boolean, - "BS": [ blob ], - "L": [ "AttributeValue" ], - "M": { "string" : "AttributeValue" }, - "N": "string", - "NS": [ "string" ], - "NULL": boolean, - "S": "string", - "SS": [ "string" ] - } -} -``` - -See AWS documentation [DynamoDB/BatchWriteItem.DeleteRequest.Key](https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_BatchWriteItem.html) - -#### Example - -```javascript -import { pipeline } from '@datastream/core' -import { awsDynamoDBDeleteItemStream } from '@datastream/aws/dynamodb' - -const streams = [ - ... - await awsDynamoDBDeleteItemStream(options) -] - -await pipeline(streams) -``` - -## Lambda - -### awsLambdaResponseStream (Readable) - - -Readable stream from the response of a Lambda invoke. - -### Options - -See AWS documentation [Lambda/InvokeWithResponseStream](https://docs.aws.amazon.com/lambda/latest/dg/API_InvokeWithResponseStream.html) - -#### Example - -```javascript -import { pipeline } from '@datastream/core' -import { awsLambdaResponseStream } from '@datastream/aws/lambda' - -const streams = [ - await awsLambdaResponseStream(options), - ... -] - -await pipeline(streams) -``` - - -## S3 - - - -### awsS3GetObjectStream (Readable) - -#### Support - -| node:stream | node:stream/web | Chrome | Edge | Firefox | Safari | Comments | -| ----------- | --------------- | ------ | ---- | ------- | ------ | -------- | -| 16.x | 18.0.0 | N/A | N/A | N/A | N/A | | - -#### Options - -#### Example - -```javascript -import { pipeline } from '@datastream/core' -import { awsS3GetObjectStream } from '@datastream/aws/s3' - -const streams = [ - await awsS3GetObjectStream(options), - ... -] - -await pipeline(streams) -``` - - -### awsS3PutObjectStream (Writable) - - -#### Support - -| node:stream | node:stream/web | Chrome | Edge | Firefox | Safari | Comments | -| ----------- | --------------- | ------ | ---- | ------- | ------ | -------- | -| 16.x | NO | N/A | N/A | N/A | N/A | | - -#### Options - -#### Example - -```javascript -import { pipeline } from '@datastream/core' -import { awsS3PutObjectStream } from '@datastream/aws/s3' - -const streams = [ - ... - await awsS3PutObjectStream(options) -] - -await pipeline(streams) -``` - -## SNS - - -### awsSNSPublishMessageStream (Writable) - - -## SQS - - -### awsSQSReceiveMessageStream (Readable) - - - - -### awsSQSDeleteMessageStream (Writable) - - - -### awsSQSSendMessageStream (Writable) - diff --git a/docs/packages/base64.md b/docs/packages/base64.md deleted file mode 100644 index 7cfef07..0000000 --- a/docs/packages/base64.md +++ /dev/null @@ -1,4 +0,0 @@ -# base64 - -## base64EncodeStream (Transform) -## base64DecodeStream (Transform) diff --git a/docs/packages/charset.md b/docs/packages/charset.md deleted file mode 100644 index 94cbd33..0000000 --- a/docs/packages/charset.md +++ /dev/null @@ -1,7 +0,0 @@ -# charset - -## charsetDetectStream (PassThrough) - -## charsetDecodeStream (Transform) - -## charsetEncodeStream (Transform) \ No newline at end of file diff --git a/docs/packages/compress.md b/docs/packages/compress.md deleted file mode 100644 index d9db13d..0000000 --- a/docs/packages/compress.md +++ /dev/null @@ -1,14 +0,0 @@ -# compress - - -## brotliCompressStream (Transform) -## brotliDecompressStream (Transform) - -## gzipCompressStream (Transform) -## gzipDecompressStream (Transform) - -## deflateCompressStream (Transform) -## deflateDecompressStream (Transform) - -## zstdCompressStream (Transform) -## zstdDecompressStream (Transform) diff --git a/docs/packages/core.md b/docs/packages/core.md deleted file mode 100644 index fbd2c18..0000000 --- a/docs/packages/core.md +++ /dev/null @@ -1,66 +0,0 @@ -# core - -## Functions -- `pipeline(stream[], streamOptions)`: Connects streams and awaits until completion. Returns results from stream taps. Will add in a terminating Writable if missing. -- `pipejoin(stream[])`: Connects streams and returns resulting stream for use with async iterators -- `result(stream)`: Run and combine streams result responses - -- `streamToArray(stream)`: Returns array from stream chunks. stream must not end with Writable. -- `streamToString(stream)`: Returns string from stream chunks. stream must not end with Writable. -- `streamToObject(stream)`: Returns object from stream chunks. stream must not end with Writable. -- `isReadable(stream)`: Return bool is stream is Readable -- `isWritable(stream)`: Return bool is stream is Writable -- `makeOptions(options)`: Make options interoperable between Readable/Writable and Transform -- `createReadableStream(input = '', streamOptions)`: Create a Readable stream from input (string, array, iterable) with options. -- `createPassThroughStream((chunk)=>{}, streamOptions)`: Create a Pass Through stream that allows observation of chunk while being passed through. -- `createTransformStream((chunk, enqueue)=>{}, streamOptions)`: Create a Transform stream that allows mutation of chunk before being passed. -- `createWritableStream((chunk)=>{}, streamOptions)`: Create a Writable stream that allows mutation of chunk before being passed. -- `tee(stream)`: -- `timeout(ms, {signal})`: setTimeout promise that can be aborted. - -- `streamOptions`: - - `highWaterMark` - - `chunkSize` - - `signal` - -## Null handling - -Node.js streams use `push(null)` to signal end-of-stream (EOF). To allow `null` values to flow through object-mode pipelines without terminating the stream, datastream wraps them with a sentinel Symbol (`Symbol.for("@datastream/null")`). - -This is handled automatically when using datastream's built-in functions (`createReadableStream`, `createTransformStream`, `createPassThroughStream`, `createWritableStream`, `streamToArray`). You only need to be aware of it when reading chunks directly from a stream, such as listening to `data` events or using `for await...of` on a mid-pipeline stream. - -```javascript -// Handled automatically - null values round-trip correctly -const output = await streamToArray(createReadableStream([1, null, 3])) -// [1, null, 3] - -// Direct access - you may see the sentinel Symbol -stream.on('data', (chunk) => { - // chunk may be Symbol.for("@datastream/null") instead of null -}) -``` - -## Examples - -```javascript -import { - pipejoin, - streamToArray, - createReadableStream, - createTransformStream -} from '@datastream/core' -import { csvParseStream } from '@datastream/csv' - -let count -const streams = [ - createReadableStream('a,b,c\r\n1,2,3'), - createTransformStream((chunk, enqueue) => { - chunk.b += 1 - enqueue(chunk) - }), - createTransformStream(console.log) -] - -const river = pipejoin(streams) -const output = await streamToArray(river) -``` diff --git a/docs/packages/csv.md b/docs/packages/csv.md deleted file mode 100644 index 095c47e..0000000 --- a/docs/packages/csv.md +++ /dev/null @@ -1,107 +0,0 @@ -# csv - -## csvParseStream (Transform) - -Takes CSV formatted string chunks and parses into flat object or array chunks. - -### Options - -| Option | Default | Description | -|---|---|---| -| `delimiterChar` | `,` | Field delimiter | -| `newlineChar` | `\r\n` | Row delimiter | -| `quoteChar` | `"` | Quote character | -| `escapeChar` | `quoteChar` | Escape character | -| `parser` | `csvQuotedParser` | Custom parser function | -| `chunkSize` | `2097152` | Buffer size before parsing begins | - -### Example - -```javascript -import { pipeline } from '@datastream/core' -import { csvParseStream } from '@datastream/csv' - -const streams = [ - ... - csvParseStream(options), - ... -] - -await pipeline(streams) -``` - -## csvFormatStream (Transform) - -Takes array chunks and outputs CSV formatted strings. Each array is formatted as one CSV row. - -### Options - -| Option | Default | Description | -|---|---|---| -| `delimiterChar` | `,` | Field delimiter | -| `newlineChar` | `\r\n` | Row delimiter | -| `quoteChar` | `"` | Quote character | -| `escapeChar` | `quoteChar` | Escape character | - -Values are automatically quoted when they contain: delimiters, newlines, quote characters, BOM, leading/trailing spaces, or formula triggers (`=`, `+`, `-`, `@`). - -### Example - -```javascript -import { pipeline } from '@datastream/core' -import { csvFormatStream, csvInjectHeaderStream, csvObjectToArray } from '@datastream/csv' - -const headers = ['a', 'b', 'c'] -const streams = [ - ... - csvObjectToArray({ headers }), - csvInjectHeaderStream({ header: headers }), - csvFormatStream(), - ... -] - -await pipeline(streams) -``` - -## csvInjectHeaderStream (Transform) - -Pushes a header array into the stream before the first data chunk. All subsequent chunks pass through unchanged. - -### Options - -| Option | Description | -|---|---| -| `header` | Array of header values to inject | - -### Example - -```javascript -import { csvInjectHeaderStream, csvFormatStream } from '@datastream/csv' - -const streams = [ - ... - csvInjectHeaderStream({ header: ['a', 'b', 'c'] }), - csvFormatStream(), - ... -] -``` - -## csvArrayToObject (Transform) - -Converts array chunks to objects using provided header keys. Wrapper around `objectFromEntriesStream`. - -### Options - -| Option | Description | -|---|---| -| `headers` | Array of key names | - -## csvObjectToArray (Transform) - -Converts object chunks to arrays using provided header keys. Wrapper around `objectToEntriesStream`. - -### Options - -| Option | Description | -|---|---| -| `headers` | Array of key names | diff --git a/docs/packages/digest.md b/docs/packages/digest.md deleted file mode 100644 index ba7ae79..0000000 --- a/docs/packages/digest.md +++ /dev/null @@ -1,3 +0,0 @@ -# digest - -## digestStream (PassThrough) \ No newline at end of file diff --git a/docs/packages/fetch.md b/docs/packages/fetch.md deleted file mode 100644 index 6db91aa..0000000 --- a/docs/packages/fetch.md +++ /dev/null @@ -1,5 +0,0 @@ -# fetch - -## fetchResponseStream (Readable) - -## fetchRequestStream (Writable) \ No newline at end of file diff --git a/docs/packages/file.md b/docs/packages/file.md deleted file mode 100644 index c97aa56..0000000 --- a/docs/packages/file.md +++ /dev/null @@ -1,4 +0,0 @@ -# file - -## fileReadStream (Readable) -## fileWriteStream (Writable) diff --git a/docs/packages/indexeddb.md b/docs/packages/indexeddb.md deleted file mode 100644 index e32bb1b..0000000 --- a/docs/packages/indexeddb.md +++ /dev/null @@ -1,4 +0,0 @@ -# indexeddb - -## indexedDBReadStream (Readable) -## indexedDBWriteStream (Writable) diff --git a/docs/packages/ipfs.md b/docs/packages/ipfs.md deleted file mode 100644 index 69487ff..0000000 --- a/docs/packages/ipfs.md +++ /dev/null @@ -1,4 +0,0 @@ -# ipfs - -## ipfsGetStream (Readable) -## ipfsAddStream (PassThrough) diff --git a/docs/packages/object.md b/docs/packages/object.md deleted file mode 100644 index 719d12c..0000000 --- a/docs/packages/object.md +++ /dev/null @@ -1,16 +0,0 @@ -# object - -## objectReadableStream (Readable) -## objectCountStream (PassThrough) -## objectBatchStream (Transform) -## objectPivotLongToWideStream (Transform) -## objectPivotWideToLongStream (Transform) -## objectKeyValueStream (Transform) -## objectKeyValuesStream (Transform) -## objectKeyJoinStream (Transform) -## objectKeyMapStream (Transform) -## objectValueMapStream (Transform) -## objectPickStream (Transform) -## objectOmitStream (Transform) -## objectFromEntriesStream (Transform) -## objectSkipConsecutiveDuplicatesStream (Transform) \ No newline at end of file diff --git a/docs/packages/string.md b/docs/packages/string.md deleted file mode 100644 index 29f2fb2..0000000 --- a/docs/packages/string.md +++ /dev/null @@ -1,8 +0,0 @@ -# string - -## stringReadableStream (Readable) -## stringLengthStream (PassThrough) -## stringCountStream (PassThrough) -## stringSkipConsecutiveDuplicates (Transform) -## stringReplaceStream (Transform) -## stringSplitStream (Transform) \ No newline at end of file diff --git a/docs/packages/validate.md b/docs/packages/validate.md deleted file mode 100644 index e344697..0000000 --- a/docs/packages/validate.md +++ /dev/null @@ -1,3 +0,0 @@ -# validate - -## validateStream (Transform) \ No newline at end of file diff --git a/docs/patterns/file.md b/docs/patterns/file.md deleted file mode 100644 index a2c530e..0000000 --- a/docs/patterns/file.md +++ /dev/null @@ -1,30 +0,0 @@ -# file - -## fileReadStream (Readable) - -```javascript -import { createReadStream } from 'node:fs' - -const fileReadableStream = (fileName, streamOptions) => { - return createReadStream(fileName) -} - -export default { - readableStream: fileReadableStream -} -``` - -## fileWriteStream (Writable) - -```javascript -import { createWriteStream } from 'node:fs' - - -export const fileWritableStream = (fileName, streamOptions) => { - return createWriteStream(fileName) -} - -export default { - writableStream: fileWritableStream -} -``` diff --git a/docs/patterns/image.md b/docs/patterns/image.md deleted file mode 100644 index 7e76faa..0000000 --- a/docs/patterns/image.md +++ /dev/null @@ -1,3 +0,0 @@ -# image - -## sharp (Transform) \ No newline at end of file diff --git a/docs/patterns/sql.md b/docs/patterns/sql.md deleted file mode 100644 index 437ea55..0000000 --- a/docs/patterns/sql.md +++ /dev/null @@ -1,9 +0,0 @@ -# sql - -## pgCopyToStream (Readable) -## pgCopuFromStream (Writable) - -## postgresCopyToStream (Readable) -## postgresCopyFromStream (Writable) - - diff --git a/package-lock.json b/package-lock.json index 352c49b..264fae7 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "@datastream/monorepo", - "version": "0.3.0", + "version": "0.3.1", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "@datastream/monorepo", - "version": "0.3.0", + "version": "0.3.1", "license": "MIT", "workspaces": [ "packages/*", @@ -30,7 +30,7 @@ }, ".github": { "name": "@datastream/github-workflows", - "version": "0.2.0", + "version": "0.3.1", "devDependencies": { "license-check-and-add": "4.0.5", "lockfile-lint": "5.0.0" @@ -3832,17 +3832,17 @@ } }, "node_modules/@smithy/config-resolver": { - "version": "4.4.15", - "resolved": "https://registry.npmjs.org/@smithy/config-resolver/-/config-resolver-4.4.15.tgz", - "integrity": "sha512-BJdMBY5YO9iHh+lPLYdHv6LbX+J8IcPCYMl1IJdBt2KDWNHwONHrPVHk3ttYBqJd9wxv84wlbN0f7GlQzcQtNQ==", + "version": "4.4.16", + "resolved": "https://registry.npmjs.org/@smithy/config-resolver/-/config-resolver-4.4.16.tgz", + "integrity": "sha512-GFlGPNLZKrGfqWpqVb31z7hvYCA9ZscfX1buYnvvMGcRYsQQnhH+4uN6mWWflcD5jB4OXP/LBrdpukEdjl41tg==", "dev": true, "license": "Apache-2.0", "dependencies": { - "@smithy/node-config-provider": "^4.3.13", - "@smithy/types": "^4.14.0", + "@smithy/node-config-provider": "^4.3.14", + "@smithy/types": "^4.14.1", "@smithy/util-config-provider": "^4.2.2", - "@smithy/util-endpoints": "^3.4.0", - "@smithy/util-middleware": "^4.2.13", + "@smithy/util-endpoints": "^3.4.1", + "@smithy/util-middleware": "^4.2.14", "tslib": "^2.6.2" }, "engines": { @@ -3850,19 +3850,19 @@ } }, "node_modules/@smithy/core": { - "version": "3.23.14", - "resolved": "https://registry.npmjs.org/@smithy/core/-/core-3.23.14.tgz", - "integrity": "sha512-vJ0IhpZxZAkFYOegMKSrxw7ujhhT2pass/1UEcZ4kfl5srTAqtPU5I7MdYQoreVas3204ykCiNhY1o7Xlz6Yyg==", + "version": "3.23.15", + "resolved": "https://registry.npmjs.org/@smithy/core/-/core-3.23.15.tgz", + "integrity": "sha512-E7GVCgsQttzfujEZb6Qep005wWf4xiL4x06apFEtzQMWYBPggZh/0cnOxPficw5cuK/YjjkehKoIN4YUaSh0UQ==", "dev": true, "license": "Apache-2.0", "dependencies": { - "@smithy/protocol-http": "^5.3.13", - "@smithy/types": "^4.14.0", - "@smithy/url-parser": "^4.2.13", + "@smithy/protocol-http": "^5.3.14", + "@smithy/types": "^4.14.1", + "@smithy/url-parser": "^4.2.14", "@smithy/util-base64": "^4.3.2", "@smithy/util-body-length-browser": "^4.2.2", - "@smithy/util-middleware": "^4.2.13", - "@smithy/util-stream": "^4.5.22", + "@smithy/util-middleware": "^4.2.14", + "@smithy/util-stream": "^4.5.23", "@smithy/util-utf8": "^4.2.2", "@smithy/uuid": "^1.1.2", "tslib": "^2.6.2" @@ -3872,16 +3872,16 @@ } }, "node_modules/@smithy/credential-provider-imds": { - "version": "4.2.13", - "resolved": "https://registry.npmjs.org/@smithy/credential-provider-imds/-/credential-provider-imds-4.2.13.tgz", - "integrity": "sha512-wboCPijzf6RJKLOvnjDAiBxGSmSnGXj35o5ZAWKDaHa/cvQ5U3ZJ13D4tMCE8JG4dxVAZFy/P0x/V9CwwdfULQ==", + "version": "4.2.14", + "resolved": "https://registry.npmjs.org/@smithy/credential-provider-imds/-/credential-provider-imds-4.2.14.tgz", + "integrity": "sha512-Au28zBN48ZAoXdooGUHemuVBrkE+Ie6RPmGNIAJsFqj33Vhb6xAgRifUydZ2aY+M+KaMAETAlKk5NC5h1G7wpg==", "dev": true, "license": "Apache-2.0", "dependencies": { - "@smithy/node-config-provider": "^4.3.13", - "@smithy/property-provider": "^4.2.13", - "@smithy/types": "^4.14.0", - "@smithy/url-parser": "^4.2.13", + "@smithy/node-config-provider": "^4.3.14", + "@smithy/property-provider": "^4.2.14", + "@smithy/types": "^4.14.1", + "@smithy/url-parser": "^4.2.14", "tslib": "^2.6.2" }, "engines": { @@ -3889,14 +3889,14 @@ } }, "node_modules/@smithy/eventstream-codec": { - "version": "4.2.13", - "resolved": "https://registry.npmjs.org/@smithy/eventstream-codec/-/eventstream-codec-4.2.13.tgz", - "integrity": "sha512-vYahwBAtRaAcFbOmE9aLr12z7RiHYDSLcnogSdxfm7kKfsNa3wH+NU5r7vTeB5rKvLsWyPjVX8iH94brP7umiQ==", + "version": "4.2.14", + "resolved": "https://registry.npmjs.org/@smithy/eventstream-codec/-/eventstream-codec-4.2.14.tgz", + "integrity": "sha512-erZq0nOIpzfeZdCyzZjdJb4nVSKLUmSkaQUVkRGQTXs30gyUGeKnrYEg+Xe1W5gE3aReS7IgsvANwVPxSzY6Pw==", "dev": true, "license": "Apache-2.0", "dependencies": { "@aws-crypto/crc32": "5.2.0", - "@smithy/types": "^4.14.0", + "@smithy/types": "^4.14.1", "@smithy/util-hex-encoding": "^4.2.2", "tslib": "^2.6.2" }, @@ -3905,14 +3905,14 @@ } }, "node_modules/@smithy/eventstream-serde-browser": { - "version": "4.2.13", - "resolved": "https://registry.npmjs.org/@smithy/eventstream-serde-browser/-/eventstream-serde-browser-4.2.13.tgz", - "integrity": "sha512-wwybfcOX0tLqCcBP378TIU9IqrDuZq/tDV48LlZNydMpCnqnYr+hWBAYbRE+rFFf/p7IkDJySM3bgiMKP2ihPg==", + "version": "4.2.14", + "resolved": "https://registry.npmjs.org/@smithy/eventstream-serde-browser/-/eventstream-serde-browser-4.2.14.tgz", + "integrity": "sha512-8IelTCtTctWRbb+0Dcy+C0aICh1qa0qWXqgjcXDmMuCvPJRnv26hiDZoAau2ILOniki65mCPKqOQs/BaWvO4CQ==", "dev": true, "license": "Apache-2.0", "dependencies": { - "@smithy/eventstream-serde-universal": "^4.2.13", - "@smithy/types": "^4.14.0", + "@smithy/eventstream-serde-universal": "^4.2.14", + "@smithy/types": "^4.14.1", "tslib": "^2.6.2" }, "engines": { @@ -3920,13 +3920,13 @@ } }, "node_modules/@smithy/eventstream-serde-config-resolver": { - "version": "4.3.13", - "resolved": "https://registry.npmjs.org/@smithy/eventstream-serde-config-resolver/-/eventstream-serde-config-resolver-4.3.13.tgz", - "integrity": "sha512-ied1lO559PtAsMJzg2TKRlctLnEi1PfkNeMMpdwXDImk1zV9uvS/Oxoy/vcy9uv1GKZAjDAB5xT6ziE9fzm5wA==", + "version": "4.3.14", + "resolved": "https://registry.npmjs.org/@smithy/eventstream-serde-config-resolver/-/eventstream-serde-config-resolver-4.3.14.tgz", + "integrity": "sha512-sqHiHpYRYo3FJlaIxD1J8PhbcmJAm7IuM16mVnwSkCToD7g00IBZzKuiLNMGmftULmEUX6/UAz8/NN5uMP8bVA==", "dev": true, "license": "Apache-2.0", "dependencies": { - "@smithy/types": "^4.14.0", + "@smithy/types": "^4.14.1", "tslib": "^2.6.2" }, "engines": { @@ -3934,14 +3934,14 @@ } }, "node_modules/@smithy/eventstream-serde-node": { - "version": "4.2.13", - "resolved": "https://registry.npmjs.org/@smithy/eventstream-serde-node/-/eventstream-serde-node-4.2.13.tgz", - "integrity": "sha512-hFyK+ORJrxAN3RYoaD6+gsGDQjeix8HOEkosoajvXYZ4VeqonM3G4jd9IIRm/sWGXUKmudkY9KdYjzosUqdM8A==", + "version": "4.2.14", + "resolved": "https://registry.npmjs.org/@smithy/eventstream-serde-node/-/eventstream-serde-node-4.2.14.tgz", + "integrity": "sha512-Ht/8BuGlKfFTy0H3+8eEu0vdpwGztCnaLLXtpXNdQqiR7Hj4vFScU3T436vRAjATglOIPjJXronY+1WxxNLSiw==", "dev": true, "license": "Apache-2.0", "dependencies": { - "@smithy/eventstream-serde-universal": "^4.2.13", - "@smithy/types": "^4.14.0", + "@smithy/eventstream-serde-universal": "^4.2.14", + "@smithy/types": "^4.14.1", "tslib": "^2.6.2" }, "engines": { @@ -3949,14 +3949,14 @@ } }, "node_modules/@smithy/eventstream-serde-universal": { - "version": "4.2.13", - "resolved": "https://registry.npmjs.org/@smithy/eventstream-serde-universal/-/eventstream-serde-universal-4.2.13.tgz", - "integrity": "sha512-kRrq4EKLGeOxhC2CBEhRNcu1KSzNJzYY7RK3S7CxMPgB5dRrv55WqQOtRwQxQLC04xqORFLUgnDlc6xrNUULaA==", + "version": "4.2.14", + "resolved": "https://registry.npmjs.org/@smithy/eventstream-serde-universal/-/eventstream-serde-universal-4.2.14.tgz", + "integrity": "sha512-lWyt4T2XQZUZgK3tQ3Wn0w3XBvZsK/vjTuJl6bXbnGZBHH0ZUSONTYiK9TgjTTzU54xQr3DRFwpjmhp0oLm3gg==", "dev": true, "license": "Apache-2.0", "dependencies": { - "@smithy/eventstream-codec": "^4.2.13", - "@smithy/types": "^4.14.0", + "@smithy/eventstream-codec": "^4.2.14", + "@smithy/types": "^4.14.1", "tslib": "^2.6.2" }, "engines": { @@ -3964,15 +3964,15 @@ } }, "node_modules/@smithy/fetch-http-handler": { - "version": "5.3.16", - "resolved": "https://registry.npmjs.org/@smithy/fetch-http-handler/-/fetch-http-handler-5.3.16.tgz", - "integrity": "sha512-nYDRUIvNd4mFmuXraRWt6w5UsZTNqtj4hXJA/iiOD4tuseIdLP9Lq38teH/SZTcIFCa2f+27o7hYpIsWktJKEQ==", + "version": "5.3.17", + "resolved": "https://registry.npmjs.org/@smithy/fetch-http-handler/-/fetch-http-handler-5.3.17.tgz", + "integrity": "sha512-bXOvQzaSm6MnmLaWA1elgfQcAtN4UP3vXqV97bHuoOrHQOJiLT3ds6o9eo5bqd0TJfRFpzdGnDQdW3FACiAVdw==", "dev": true, "license": "Apache-2.0", "dependencies": { - "@smithy/protocol-http": "^5.3.13", - "@smithy/querystring-builder": "^4.2.13", - "@smithy/types": "^4.14.0", + "@smithy/protocol-http": "^5.3.14", + "@smithy/querystring-builder": "^4.2.14", + "@smithy/types": "^4.14.1", "@smithy/util-base64": "^4.3.2", "tslib": "^2.6.2" }, @@ -3981,15 +3981,15 @@ } }, "node_modules/@smithy/hash-blob-browser": { - "version": "4.2.14", - "resolved": "https://registry.npmjs.org/@smithy/hash-blob-browser/-/hash-blob-browser-4.2.14.tgz", - "integrity": "sha512-rtQ5es8r/5v4rav7q5QTsfx9CtCyzrz/g7ZZZBH2xtMmd6G/KQrLOWfSHTvFOUPlVy59RQvxeBYJaLRoybMEyA==", + "version": "4.2.15", + "resolved": "https://registry.npmjs.org/@smithy/hash-blob-browser/-/hash-blob-browser-4.2.15.tgz", + "integrity": "sha512-0PJ4Al3fg2nM4qKrAIxyNcApgqHAXcBkN8FeizOz69z0rb26uZ6lMESYtxegaTlXB5Hj84JfwMPavMrwDMjucA==", "dev": true, "license": "Apache-2.0", "dependencies": { "@smithy/chunked-blob-reader": "^5.2.2", "@smithy/chunked-blob-reader-native": "^4.2.3", - "@smithy/types": "^4.14.0", + "@smithy/types": "^4.14.1", "tslib": "^2.6.2" }, "engines": { @@ -3997,13 +3997,13 @@ } }, "node_modules/@smithy/hash-node": { - "version": "4.2.13", - "resolved": "https://registry.npmjs.org/@smithy/hash-node/-/hash-node-4.2.13.tgz", - "integrity": "sha512-4/oy9h0jjmY80a2gOIo75iLl8TOPhmtx4E2Hz+PfMjvx/vLtGY4TMU/35WRyH2JHPfT5CVB38u4JRow7gnmzJA==", + "version": "4.2.14", + "resolved": "https://registry.npmjs.org/@smithy/hash-node/-/hash-node-4.2.14.tgz", + "integrity": "sha512-8ZBDY2DD4wr+GGjTpPtiglEsqr0lUP+KHqgZcWczFf6qeZ/YRjMIOoQWVQlmwu7EtxKTd8YXD8lblmYcpBIA1g==", "dev": true, "license": "Apache-2.0", "dependencies": { - "@smithy/types": "^4.14.0", + "@smithy/types": "^4.14.1", "@smithy/util-buffer-from": "^4.2.2", "@smithy/util-utf8": "^4.2.2", "tslib": "^2.6.2" @@ -4013,13 +4013,13 @@ } }, "node_modules/@smithy/hash-stream-node": { - "version": "4.2.13", - "resolved": "https://registry.npmjs.org/@smithy/hash-stream-node/-/hash-stream-node-4.2.13.tgz", - "integrity": "sha512-WdQ7HwUjINXETeh6dqUeob1UHIYx8kAn9PSp1HhM2WWegiZBYVy2WXIs1lB07SZLan/udys9SBnQGt9MQbDpdg==", + "version": "4.2.14", + "resolved": "https://registry.npmjs.org/@smithy/hash-stream-node/-/hash-stream-node-4.2.14.tgz", + "integrity": "sha512-tw4GANWkZPb6+BdD4Fgucqzey2+r73Z/GRo9zklsCdwrnxxumUV83ZIaBDdudV4Ylazw3EPTiJZhpX42105ruQ==", "dev": true, "license": "Apache-2.0", "dependencies": { - "@smithy/types": "^4.14.0", + "@smithy/types": "^4.14.1", "@smithy/util-utf8": "^4.2.2", "tslib": "^2.6.2" }, @@ -4028,13 +4028,13 @@ } }, "node_modules/@smithy/invalid-dependency": { - "version": "4.2.13", - "resolved": "https://registry.npmjs.org/@smithy/invalid-dependency/-/invalid-dependency-4.2.13.tgz", - "integrity": "sha512-jvC0RB/8BLj2SMIkY0Npl425IdnxZJxInpZJbu563zIRnVjpDMXevU3VMCRSabaLB0kf/eFIOusdGstrLJ8IDg==", + "version": "4.2.14", + "resolved": "https://registry.npmjs.org/@smithy/invalid-dependency/-/invalid-dependency-4.2.14.tgz", + "integrity": "sha512-c21qJiTSb25xvvOp+H2TNZzPCngrvl5vIPqPB8zQ/DmJF4QWXO19x1dWfMJZ6wZuuWUPPm0gV8C0cU3+ifcWuw==", "dev": true, "license": "Apache-2.0", "dependencies": { - "@smithy/types": "^4.14.0", + "@smithy/types": "^4.14.1", "tslib": "^2.6.2" }, "engines": { @@ -4055,13 +4055,13 @@ } }, "node_modules/@smithy/md5-js": { - "version": "4.2.13", - "resolved": "https://registry.npmjs.org/@smithy/md5-js/-/md5-js-4.2.13.tgz", - "integrity": "sha512-cNm7I9NXolFxtS20ojROddOEpSAeI1Obq6pd1Kj5HtHws3s9Fkk8DdHDfQSs5KuxCewZuVK6UqrJnfJmiMzDuQ==", + "version": "4.2.14", + "resolved": "https://registry.npmjs.org/@smithy/md5-js/-/md5-js-4.2.14.tgz", + "integrity": "sha512-V2v0vx+h0iUSNG1Alt+GNBMSLGCrl9iVsdd+Ap67HPM9PN479x12V8LkuMoKImNZxn3MXeuyUjls+/7ZACZghA==", "dev": true, "license": "Apache-2.0", "dependencies": { - "@smithy/types": "^4.14.0", + "@smithy/types": "^4.14.1", "@smithy/util-utf8": "^4.2.2", "tslib": "^2.6.2" }, @@ -4070,14 +4070,14 @@ } }, "node_modules/@smithy/middleware-content-length": { - "version": "4.2.13", - "resolved": "https://registry.npmjs.org/@smithy/middleware-content-length/-/middleware-content-length-4.2.13.tgz", - "integrity": "sha512-IPMLm/LE4AZwu6qiE8Rr8vJsWhs9AtOdySRXrOM7xnvclp77Tyh7hMs/FRrMf26kgIe67vFJXXOSmVxS7oKeig==", + "version": "4.2.14", + "resolved": "https://registry.npmjs.org/@smithy/middleware-content-length/-/middleware-content-length-4.2.14.tgz", + "integrity": "sha512-xhHq7fX4/3lv5NHxLUk3OeEvl0xZ+Ek3qIbWaCL4f9JwgDZEclPBElljaZCAItdGPQl/kSM4LPMOpy1MYgprpw==", "dev": true, "license": "Apache-2.0", "dependencies": { - "@smithy/protocol-http": "^5.3.13", - "@smithy/types": "^4.14.0", + "@smithy/protocol-http": "^5.3.14", + "@smithy/types": "^4.14.1", "tslib": "^2.6.2" }, "engines": { @@ -4085,19 +4085,19 @@ } }, "node_modules/@smithy/middleware-endpoint": { - "version": "4.4.29", - "resolved": "https://registry.npmjs.org/@smithy/middleware-endpoint/-/middleware-endpoint-4.4.29.tgz", - "integrity": "sha512-R9Q/58U+qBiSARGWbAbFLczECg/RmysRksX6Q8BaQEpt75I7LI6WGDZnjuC9GXSGKljEbA7N118LhGaMbfrTXw==", + "version": "4.4.30", + "resolved": "https://registry.npmjs.org/@smithy/middleware-endpoint/-/middleware-endpoint-4.4.30.tgz", + "integrity": "sha512-qS2XqhKeXmdZ4nEQ4cOxIczSP/Y91wPAHYuRwmWDCh975B7/57uxsm5d6sisnUThn2u2FwzMdJNM7AbO1YPsPg==", "dev": true, "license": "Apache-2.0", "dependencies": { - "@smithy/core": "^3.23.14", - "@smithy/middleware-serde": "^4.2.17", - "@smithy/node-config-provider": "^4.3.13", - "@smithy/shared-ini-file-loader": "^4.4.8", - "@smithy/types": "^4.14.0", - "@smithy/url-parser": "^4.2.13", - "@smithy/util-middleware": "^4.2.13", + "@smithy/core": "^3.23.15", + "@smithy/middleware-serde": "^4.2.18", + "@smithy/node-config-provider": "^4.3.14", + "@smithy/shared-ini-file-loader": "^4.4.9", + "@smithy/types": "^4.14.1", + "@smithy/url-parser": "^4.2.14", + "@smithy/util-middleware": "^4.2.14", "tslib": "^2.6.2" }, "engines": { @@ -4105,20 +4105,20 @@ } }, "node_modules/@smithy/middleware-retry": { - "version": "4.5.1", - "resolved": "https://registry.npmjs.org/@smithy/middleware-retry/-/middleware-retry-4.5.1.tgz", - "integrity": "sha512-/zY+Gp7Qj2D2hVm3irkCyONER7E9MiX3cUUm/k2ZmhkzZkrPgwVS4aJ5NriZUEN/M0D1hhjrgjUmX04HhRwdWA==", + "version": "4.5.3", + "resolved": "https://registry.npmjs.org/@smithy/middleware-retry/-/middleware-retry-4.5.3.tgz", + "integrity": "sha512-TE8dJNi6JuxzGSxMCVd3i9IEWDndCl3bmluLsBNDWok8olgj65OfkndMhl9SZ7m14c+C5SQn/PcUmrDl57rSFw==", "dev": true, "license": "Apache-2.0", "dependencies": { - "@smithy/core": "^3.23.14", - "@smithy/node-config-provider": "^4.3.13", - "@smithy/protocol-http": "^5.3.13", - "@smithy/service-error-classification": "^4.2.13", - "@smithy/smithy-client": "^4.12.9", - "@smithy/types": "^4.14.0", - "@smithy/util-middleware": "^4.2.13", - "@smithy/util-retry": "^4.3.1", + "@smithy/core": "^3.23.15", + "@smithy/node-config-provider": "^4.3.14", + "@smithy/protocol-http": "^5.3.14", + "@smithy/service-error-classification": "^4.2.14", + "@smithy/smithy-client": "^4.12.11", + "@smithy/types": "^4.14.1", + "@smithy/util-middleware": "^4.2.14", + "@smithy/util-retry": "^4.3.2", "@smithy/uuid": "^1.1.2", "tslib": "^2.6.2" }, @@ -4127,15 +4127,15 @@ } }, "node_modules/@smithy/middleware-serde": { - "version": "4.2.17", - "resolved": "https://registry.npmjs.org/@smithy/middleware-serde/-/middleware-serde-4.2.17.tgz", - "integrity": "sha512-0T2mcaM6v9W1xku86Dk0bEW7aEseG6KenFkPK98XNw0ZhOqOiD1MrMsdnQw9QsL3/Oa85T53iSMlm0SZdSuIEQ==", + "version": "4.2.18", + "resolved": "https://registry.npmjs.org/@smithy/middleware-serde/-/middleware-serde-4.2.18.tgz", + "integrity": "sha512-M6CSgnp3v4tYz9ynj2JHbA60woBZcGqEwNjTKjBsNHPV26R1ZX52+0wW8WsZU18q45jD0tw2wL22S17Ze9LpEw==", "dev": true, "license": "Apache-2.0", "dependencies": { - "@smithy/core": "^3.23.14", - "@smithy/protocol-http": "^5.3.13", - "@smithy/types": "^4.14.0", + "@smithy/core": "^3.23.15", + "@smithy/protocol-http": "^5.3.14", + "@smithy/types": "^4.14.1", "tslib": "^2.6.2" }, "engines": { @@ -4143,13 +4143,13 @@ } }, "node_modules/@smithy/middleware-stack": { - "version": "4.2.13", - "resolved": "https://registry.npmjs.org/@smithy/middleware-stack/-/middleware-stack-4.2.13.tgz", - "integrity": "sha512-g72jN/sGDLyTanrCLH9fhg3oysO3f7tQa6eWWsMyn2BiYNCgjF24n4/I9wff/5XidFvjj9ilipAoQrurTUrLvw==", + "version": "4.2.14", + "resolved": "https://registry.npmjs.org/@smithy/middleware-stack/-/middleware-stack-4.2.14.tgz", + "integrity": "sha512-2dvkUKLuFdKsCRmOE4Mn63co0Djtsm+JMh0bYZQupN1pJwMeE8FmQmRLLzzEMN0dnNi7CDCYYH8F0EVwWiPBeA==", "dev": true, "license": "Apache-2.0", "dependencies": { - "@smithy/types": "^4.14.0", + "@smithy/types": "^4.14.1", "tslib": "^2.6.2" }, "engines": { @@ -4157,15 +4157,15 @@ } }, "node_modules/@smithy/node-config-provider": { - "version": "4.3.13", - "resolved": "https://registry.npmjs.org/@smithy/node-config-provider/-/node-config-provider-4.3.13.tgz", - "integrity": "sha512-iGxQ04DsKXLckbgnX4ipElrOTk+IHgTyu0q0WssZfYhDm9CQWHmu6cOeI5wmWRxpXbBDhIIfXMWz5tPEtcVqbw==", + "version": "4.3.14", + "resolved": "https://registry.npmjs.org/@smithy/node-config-provider/-/node-config-provider-4.3.14.tgz", + "integrity": "sha512-S+gFjyo/weSVL0P1b9Ts8C/CwIfNCgUPikk3sl6QVsfE/uUuO+QsF+NsE/JkpvWqqyz1wg7HFdiaZuj5CoBMRg==", "dev": true, "license": "Apache-2.0", "dependencies": { - "@smithy/property-provider": "^4.2.13", - "@smithy/shared-ini-file-loader": "^4.4.8", - "@smithy/types": "^4.14.0", + "@smithy/property-provider": "^4.2.14", + "@smithy/shared-ini-file-loader": "^4.4.9", + "@smithy/types": "^4.14.1", "tslib": "^2.6.2" }, "engines": { @@ -4173,15 +4173,15 @@ } }, "node_modules/@smithy/node-http-handler": { - "version": "4.5.2", - "resolved": "https://registry.npmjs.org/@smithy/node-http-handler/-/node-http-handler-4.5.2.tgz", - "integrity": "sha512-/oD7u8M0oj2ZTFw7GkuuHWpIxtWdLlnyNkbrWcyVYhd5RJNDuczdkb0wfnQICyNFrVPlr8YHOhamjNy3zidhmA==", + "version": "4.5.3", + "resolved": "https://registry.npmjs.org/@smithy/node-http-handler/-/node-http-handler-4.5.3.tgz", + "integrity": "sha512-lc5jFL++x17sPhIwMWJ3YOnqmSjw/2Po6VLDlUIXvxVWRuJwRXnJ4jOBBLB0cfI5BB5ehIl02Fxr1PDvk/kxDw==", "dev": true, "license": "Apache-2.0", "dependencies": { - "@smithy/protocol-http": "^5.3.13", - "@smithy/querystring-builder": "^4.2.13", - "@smithy/types": "^4.14.0", + "@smithy/protocol-http": "^5.3.14", + "@smithy/querystring-builder": "^4.2.14", + "@smithy/types": "^4.14.1", "tslib": "^2.6.2" }, "engines": { @@ -4189,13 +4189,13 @@ } }, "node_modules/@smithy/property-provider": { - "version": "4.2.13", - "resolved": "https://registry.npmjs.org/@smithy/property-provider/-/property-provider-4.2.13.tgz", - "integrity": "sha512-bGzUCthxRmezuxkbu9wD33wWg9KX3hJpCXpQ93vVkPrHn9ZW6KNNdY5xAUWNuRCwQ+VyboFuWirG1lZhhkcyRQ==", + "version": "4.2.14", + "resolved": "https://registry.npmjs.org/@smithy/property-provider/-/property-provider-4.2.14.tgz", + "integrity": "sha512-WuM31CgfsnQ/10i7NYr0PyxqknD72Y5uMfUMVSniPjbEPceiTErb4eIqJQ+pdxNEAUEWrewrGjIRjVbVHsxZiQ==", "dev": true, "license": "Apache-2.0", "dependencies": { - "@smithy/types": "^4.14.0", + "@smithy/types": "^4.14.1", "tslib": "^2.6.2" }, "engines": { @@ -4203,13 +4203,13 @@ } }, "node_modules/@smithy/protocol-http": { - "version": "5.3.13", - "resolved": "https://registry.npmjs.org/@smithy/protocol-http/-/protocol-http-5.3.13.tgz", - "integrity": "sha512-+HsmuJUF4u8POo6s8/a2Yb/AQ5t/YgLovCuHF9oxbocqv+SZ6gd8lC2duBFiCA/vFHoHQhoq7QjqJqZC6xOxxg==", + "version": "5.3.14", + "resolved": "https://registry.npmjs.org/@smithy/protocol-http/-/protocol-http-5.3.14.tgz", + "integrity": "sha512-dN5F8kHx8RNU0r+pCwNmFZyz6ChjMkzShy/zup6MtkRmmix4vZzJdW+di7x//b1LiynIev88FM18ie+wwPcQtQ==", "dev": true, "license": "Apache-2.0", "dependencies": { - "@smithy/types": "^4.14.0", + "@smithy/types": "^4.14.1", "tslib": "^2.6.2" }, "engines": { @@ -4217,13 +4217,13 @@ } }, "node_modules/@smithy/querystring-builder": { - "version": "4.2.13", - "resolved": "https://registry.npmjs.org/@smithy/querystring-builder/-/querystring-builder-4.2.13.tgz", - "integrity": "sha512-tG4aOYFCZdPMjbgfhnIQ322H//ojujldp1SrHPHpBSb3NqgUp3dwiUGRJzie87hS1DYwWGqDuPaowoDF+rYCbQ==", + "version": "4.2.14", + "resolved": "https://registry.npmjs.org/@smithy/querystring-builder/-/querystring-builder-4.2.14.tgz", + "integrity": "sha512-XYA5Z0IqTeF+5XDdh4BBmSA0HvbgVZIyv4cmOoUheDNR57K1HgBp9ukUMx3Cr3XpDHHpLBnexPE3LAtDsZkj2A==", "dev": true, "license": "Apache-2.0", "dependencies": { - "@smithy/types": "^4.14.0", + "@smithy/types": "^4.14.1", "@smithy/util-uri-escape": "^4.2.2", "tslib": "^2.6.2" }, @@ -4232,13 +4232,13 @@ } }, "node_modules/@smithy/querystring-parser": { - "version": "4.2.13", - "resolved": "https://registry.npmjs.org/@smithy/querystring-parser/-/querystring-parser-4.2.13.tgz", - "integrity": "sha512-hqW3Q4P+CDzUyQ87GrboGMeD7XYNMOF+CuTwu936UQRB/zeYn3jys8C3w+wMkDfY7CyyyVwZQ5cNFoG0x1pYmA==", + "version": "4.2.14", + "resolved": "https://registry.npmjs.org/@smithy/querystring-parser/-/querystring-parser-4.2.14.tgz", + "integrity": "sha512-hr+YyqBD23GVvRxGGrcc/oOeNlK3PzT5Fu4dzrDXxzS1LpFiuL2PQQqKPs87M79aW7ziMs+nvB3qdw77SqE7Lw==", "dev": true, "license": "Apache-2.0", "dependencies": { - "@smithy/types": "^4.14.0", + "@smithy/types": "^4.14.1", "tslib": "^2.6.2" }, "engines": { @@ -4246,26 +4246,26 @@ } }, "node_modules/@smithy/service-error-classification": { - "version": "4.2.13", - "resolved": "https://registry.npmjs.org/@smithy/service-error-classification/-/service-error-classification-4.2.13.tgz", - "integrity": "sha512-a0s8XZMfOC/qpqq7RCPvJlk93rWFrElH6O++8WJKz0FqnA4Y7fkNi/0mnGgSH1C4x6MFsuBA8VKu4zxFrMe5Vw==", + "version": "4.2.14", + "resolved": "https://registry.npmjs.org/@smithy/service-error-classification/-/service-error-classification-4.2.14.tgz", + "integrity": "sha512-vVimoUnGxlx4eLLQbZImdOZFOe+Zh+5ACntv8VxZuGP72LdWu5GV3oEmCahSEReBgRJoWjypFkrehSj7BWx1HQ==", "dev": true, "license": "Apache-2.0", "dependencies": { - "@smithy/types": "^4.14.0" + "@smithy/types": "^4.14.1" }, "engines": { "node": ">=18.0.0" } }, "node_modules/@smithy/shared-ini-file-loader": { - "version": "4.4.8", - "resolved": "https://registry.npmjs.org/@smithy/shared-ini-file-loader/-/shared-ini-file-loader-4.4.8.tgz", - "integrity": "sha512-VZCZx2bZasxdqxVgEAhREvDSlkatTPnkdWy1+Kiy8w7kYPBosW0V5IeDwzDUMvWBt56zpK658rx1cOBFOYaPaw==", + "version": "4.4.9", + "resolved": "https://registry.npmjs.org/@smithy/shared-ini-file-loader/-/shared-ini-file-loader-4.4.9.tgz", + "integrity": "sha512-495/V2I15SHgedSJoDPD23JuSfKAp726ZI1V0wtjB07Wh7q/0tri/0e0DLefZCHgxZonrGKt/OCTpAtP1wE1kQ==", "dev": true, "license": "Apache-2.0", "dependencies": { - "@smithy/types": "^4.14.0", + "@smithy/types": "^4.14.1", "tslib": "^2.6.2" }, "engines": { @@ -4273,17 +4273,17 @@ } }, "node_modules/@smithy/signature-v4": { - "version": "5.3.13", - "resolved": "https://registry.npmjs.org/@smithy/signature-v4/-/signature-v4-5.3.13.tgz", - "integrity": "sha512-YpYSyM0vMDwKbHD/JA7bVOF6kToVRpa+FM5ateEVRpsTNu564g1muBlkTubXhSKKYXInhpADF46FPyrZcTLpXg==", + "version": "5.3.14", + "resolved": "https://registry.npmjs.org/@smithy/signature-v4/-/signature-v4-5.3.14.tgz", + "integrity": "sha512-1D9Y/nmlVjCeSivCbhZ7hgEpmHyY1h0GvpSZt3l0xcD9JjmjVC1CHOozS6+Gh+/ldMH8JuJ6cujObQqfayAVFA==", "dev": true, "license": "Apache-2.0", "dependencies": { "@smithy/is-array-buffer": "^4.2.2", - "@smithy/protocol-http": "^5.3.13", - "@smithy/types": "^4.14.0", + "@smithy/protocol-http": "^5.3.14", + "@smithy/types": "^4.14.1", "@smithy/util-hex-encoding": "^4.2.2", - "@smithy/util-middleware": "^4.2.13", + "@smithy/util-middleware": "^4.2.14", "@smithy/util-uri-escape": "^4.2.2", "@smithy/util-utf8": "^4.2.2", "tslib": "^2.6.2" @@ -4293,18 +4293,18 @@ } }, "node_modules/@smithy/smithy-client": { - "version": "4.12.9", - "resolved": "https://registry.npmjs.org/@smithy/smithy-client/-/smithy-client-4.12.9.tgz", - "integrity": "sha512-ovaLEcTU5olSeHcRXcxV6viaKtpkHZumn6Ps0yn7dRf2rRSfy794vpjOtrWDO0d1auDSvAqxO+lyhERSXQ03EQ==", + "version": "4.12.11", + "resolved": "https://registry.npmjs.org/@smithy/smithy-client/-/smithy-client-4.12.11.tgz", + "integrity": "sha512-wzz/Wa1CH/Tlhxh0s4DQPEcXSxSVfJ59AZcUh9Gu0c6JTlKuwGf4o/3P2TExv0VbtPFt8odIBG+eQGK2+vTECg==", "dev": true, "license": "Apache-2.0", "dependencies": { - "@smithy/core": "^3.23.14", - "@smithy/middleware-endpoint": "^4.4.29", - "@smithy/middleware-stack": "^4.2.13", - "@smithy/protocol-http": "^5.3.13", - "@smithy/types": "^4.14.0", - "@smithy/util-stream": "^4.5.22", + "@smithy/core": "^3.23.15", + "@smithy/middleware-endpoint": "^4.4.30", + "@smithy/middleware-stack": "^4.2.14", + "@smithy/protocol-http": "^5.3.14", + "@smithy/types": "^4.14.1", + "@smithy/util-stream": "^4.5.23", "tslib": "^2.6.2" }, "engines": { @@ -4312,9 +4312,9 @@ } }, "node_modules/@smithy/types": { - "version": "4.14.0", - "resolved": "https://registry.npmjs.org/@smithy/types/-/types-4.14.0.tgz", - "integrity": "sha512-OWgntFLW88kx2qvf/c/67Vno1yuXm/f9M7QFAtVkkO29IJXGBIg0ycEaBTH0kvCtwmvZxRujrgP5a86RvsXJAQ==", + "version": "4.14.1", + "resolved": "https://registry.npmjs.org/@smithy/types/-/types-4.14.1.tgz", + "integrity": "sha512-59b5HtSVrVR/eYNei3BUj3DCPKD/G7EtDDe7OEJE7i7FtQFugYo6MxbotS8mVJkLNVf8gYaAlEBwwtJ9HzhWSg==", "dev": true, "license": "Apache-2.0", "dependencies": { @@ -4325,14 +4325,14 @@ } }, "node_modules/@smithy/url-parser": { - "version": "4.2.13", - "resolved": "https://registry.npmjs.org/@smithy/url-parser/-/url-parser-4.2.13.tgz", - "integrity": "sha512-2G03yoboIRZlZze2+PT4GZEjgwQsJjUgn6iTsvxA02bVceHR6vp4Cuk7TUnPFWKF+ffNUk3kj4COwkENS2K3vw==", + "version": "4.2.14", + "resolved": "https://registry.npmjs.org/@smithy/url-parser/-/url-parser-4.2.14.tgz", + "integrity": "sha512-p06BiBigJ8bTA3MgnOfCtDUWnAMY0YfedO/GRpmc7p+wg3KW8vbXy1xwSu5ASy0wV7rRYtlfZOIKH4XqfhjSQQ==", "dev": true, "license": "Apache-2.0", "dependencies": { - "@smithy/querystring-parser": "^4.2.13", - "@smithy/types": "^4.14.0", + "@smithy/querystring-parser": "^4.2.14", + "@smithy/types": "^4.14.1", "tslib": "^2.6.2" }, "engines": { @@ -4408,15 +4408,15 @@ } }, "node_modules/@smithy/util-defaults-mode-browser": { - "version": "4.3.45", - "resolved": "https://registry.npmjs.org/@smithy/util-defaults-mode-browser/-/util-defaults-mode-browser-4.3.45.tgz", - "integrity": "sha512-ag9sWc6/nWZAuK3Wm9KlFJUnRkXLrXn33RFjIAmCTFThqLHY+7wCst10BGq56FxslsDrjhSie46c8OULS+BiIw==", + "version": "4.3.47", + "resolved": "https://registry.npmjs.org/@smithy/util-defaults-mode-browser/-/util-defaults-mode-browser-4.3.47.tgz", + "integrity": "sha512-zlIuXai3/SHjQUQ8y3g/woLvrH573SK2wNjcDaHu5e9VOcC0JwM1MI0Sq0GZJyN3BwSUneIhpjZ18nsiz5AtQw==", "dev": true, "license": "Apache-2.0", "dependencies": { - "@smithy/property-provider": "^4.2.13", - "@smithy/smithy-client": "^4.12.9", - "@smithy/types": "^4.14.0", + "@smithy/property-provider": "^4.2.14", + "@smithy/smithy-client": "^4.12.11", + "@smithy/types": "^4.14.1", "tslib": "^2.6.2" }, "engines": { @@ -4424,18 +4424,18 @@ } }, "node_modules/@smithy/util-defaults-mode-node": { - "version": "4.2.50", - "resolved": "https://registry.npmjs.org/@smithy/util-defaults-mode-node/-/util-defaults-mode-node-4.2.50.tgz", - "integrity": "sha512-xpjncL5XozFA3No7WypTsPU1du0fFS8flIyO+Wh2nhCy7bpEapvU7BR55Bg+wrfw+1cRA+8G8UsTjaxgzrMzXg==", + "version": "4.2.52", + "resolved": "https://registry.npmjs.org/@smithy/util-defaults-mode-node/-/util-defaults-mode-node-4.2.52.tgz", + "integrity": "sha512-cQBz8g68Vnw1W2meXlkb3D/hXJU+Taiyj9P8qLJtjREEV9/Td65xi4A/H1sRQ8EIgX5qbZbvdYPKygKLholZ3w==", "dev": true, "license": "Apache-2.0", "dependencies": { - "@smithy/config-resolver": "^4.4.15", - "@smithy/credential-provider-imds": "^4.2.13", - "@smithy/node-config-provider": "^4.3.13", - "@smithy/property-provider": "^4.2.13", - "@smithy/smithy-client": "^4.12.9", - "@smithy/types": "^4.14.0", + "@smithy/config-resolver": "^4.4.16", + "@smithy/credential-provider-imds": "^4.2.14", + "@smithy/node-config-provider": "^4.3.14", + "@smithy/property-provider": "^4.2.14", + "@smithy/smithy-client": "^4.12.11", + "@smithy/types": "^4.14.1", "tslib": "^2.6.2" }, "engines": { @@ -4443,14 +4443,14 @@ } }, "node_modules/@smithy/util-endpoints": { - "version": "3.4.0", - "resolved": "https://registry.npmjs.org/@smithy/util-endpoints/-/util-endpoints-3.4.0.tgz", - "integrity": "sha512-QQHGPKkw6NPcU6TJ1rNEEa201srPtZiX4k61xL163vvs9sTqW/XKz+UEuJ00uvPqoN+5Rs4Ka1UJ7+Mp03IXJw==", + "version": "3.4.1", + "resolved": "https://registry.npmjs.org/@smithy/util-endpoints/-/util-endpoints-3.4.1.tgz", + "integrity": "sha512-wMxNDZJrgS5mQV9oxCs4TWl5767VMgOfqfZ3JHyCkMtGC2ykW9iPqMvFur695Otcc5yxLG8OKO/80tsQBxrhXg==", "dev": true, "license": "Apache-2.0", "dependencies": { - "@smithy/node-config-provider": "^4.3.13", - "@smithy/types": "^4.14.0", + "@smithy/node-config-provider": "^4.3.14", + "@smithy/types": "^4.14.1", "tslib": "^2.6.2" }, "engines": { @@ -4471,13 +4471,13 @@ } }, "node_modules/@smithy/util-middleware": { - "version": "4.2.13", - "resolved": "https://registry.npmjs.org/@smithy/util-middleware/-/util-middleware-4.2.13.tgz", - "integrity": "sha512-GTooyrlmRTqvUen4eK7/K1p6kryF7bnDfq6XsAbIsf2mo51B/utaH+XThY6dKgNCWzMAaH/+OLmqaBuLhLWRow==", + "version": "4.2.14", + "resolved": "https://registry.npmjs.org/@smithy/util-middleware/-/util-middleware-4.2.14.tgz", + "integrity": "sha512-1Su2vj9RYNDEv/V+2E+jXkkwGsgR7dc4sfHn9Z7ruzQHJIEni9zzw5CauvRXlFJfmgcqYP8fWa0dkh2Q2YaQyw==", "dev": true, "license": "Apache-2.0", "dependencies": { - "@smithy/types": "^4.14.0", + "@smithy/types": "^4.14.1", "tslib": "^2.6.2" }, "engines": { @@ -4485,14 +4485,14 @@ } }, "node_modules/@smithy/util-retry": { - "version": "4.3.1", - "resolved": "https://registry.npmjs.org/@smithy/util-retry/-/util-retry-4.3.1.tgz", - "integrity": "sha512-FwmicpgWOkP5kZUjN3y+3JIom8NLGqSAJBeoIgK0rIToI817TEBHCrd0A2qGeKQlgDeP+Jzn4i0H/NLAXGy9uQ==", + "version": "4.3.2", + "resolved": "https://registry.npmjs.org/@smithy/util-retry/-/util-retry-4.3.2.tgz", + "integrity": "sha512-2+KTsJEwTi63NUv4uR9IQ+IFT1yu6Rf6JuoBK2WKaaJ/TRvOiOVGcXAsEqX/TQN2thR9yII21kPUJq1UV/WI2A==", "dev": true, "license": "Apache-2.0", "dependencies": { - "@smithy/service-error-classification": "^4.2.13", - "@smithy/types": "^4.14.0", + "@smithy/service-error-classification": "^4.2.14", + "@smithy/types": "^4.14.1", "tslib": "^2.6.2" }, "engines": { @@ -4500,15 +4500,15 @@ } }, "node_modules/@smithy/util-stream": { - "version": "4.5.22", - "resolved": "https://registry.npmjs.org/@smithy/util-stream/-/util-stream-4.5.22.tgz", - "integrity": "sha512-3H8iq/0BfQjUs2/4fbHZ9aG9yNzcuZs24LPkcX1Q7Z+qpqaGM8+qbGmE8zo9m2nCRgamyvS98cHdcWvR6YUsew==", + "version": "4.5.23", + "resolved": "https://registry.npmjs.org/@smithy/util-stream/-/util-stream-4.5.23.tgz", + "integrity": "sha512-N6on1+ngJ3RznZOnDWNveIwnTSlqxNnXuNAh7ez889ZZaRdXoNRTXKgmYOLe6dB0gCmAVtuRScE1hymQFl4hpg==", "dev": true, "license": "Apache-2.0", "dependencies": { - "@smithy/fetch-http-handler": "^5.3.16", - "@smithy/node-http-handler": "^4.5.2", - "@smithy/types": "^4.14.0", + "@smithy/fetch-http-handler": "^5.3.17", + "@smithy/node-http-handler": "^4.5.3", + "@smithy/types": "^4.14.1", "@smithy/util-base64": "^4.3.2", "@smithy/util-buffer-from": "^4.2.2", "@smithy/util-hex-encoding": "^4.2.2", @@ -4547,13 +4547,13 @@ } }, "node_modules/@smithy/util-waiter": { - "version": "4.2.15", - "resolved": "https://registry.npmjs.org/@smithy/util-waiter/-/util-waiter-4.2.15.tgz", - "integrity": "sha512-oUt9o7n8hBv3BL56sLSneL0XeigZSuem0Hr78JaoK33D9oKieyCvVP8eTSe3j7g2mm/S1DvzxKieG7JEWNJUNg==", + "version": "4.2.16", + "resolved": "https://registry.npmjs.org/@smithy/util-waiter/-/util-waiter-4.2.16.tgz", + "integrity": "sha512-GtclrKoZ3Lt7jPQ7aTIYKfjY92OgceScftVnkTsG8e1KV8rkvZgN+ny6YSRhd9hxB8rZtwVbmln7NTvE5O3GmQ==", "dev": true, "license": "Apache-2.0", "dependencies": { - "@smithy/types": "^4.14.0", + "@smithy/types": "^4.14.1", "tslib": "^2.6.2" }, "engines": { @@ -10005,10 +10005,10 @@ }, "packages/aws": { "name": "@datastream/aws", - "version": "0.3.0", + "version": "0.3.1", "license": "MIT", "dependencies": { - "@datastream/core": "0.3.0" + "@datastream/core": "0.3.1" }, "devDependencies": { "@aws-sdk/client-cloudwatch-logs": "^3.0.0", @@ -10068,10 +10068,10 @@ }, "packages/base64": { "name": "@datastream/base64", - "version": "0.3.0", + "version": "0.3.1", "license": "MIT", "dependencies": { - "@datastream/core": "0.3.0" + "@datastream/core": "0.3.1" }, "engines": { "node": ">=24" @@ -10079,10 +10079,10 @@ }, "packages/charset": { "name": "@datastream/charset", - "version": "0.3.0", + "version": "0.3.1", "license": "MIT", "dependencies": { - "@datastream/core": "0.3.0", + "@datastream/core": "0.3.1", "charset-detector": "0.0.2", "iconv-lite": "0.7.2" }, @@ -10092,10 +10092,10 @@ }, "packages/compress": { "name": "@datastream/compress", - "version": "0.3.0", + "version": "0.3.1", "license": "MIT", "dependencies": { - "@datastream/core": "0.3.0" + "@datastream/core": "0.3.1" }, "engines": { "node": ">=24" @@ -10119,10 +10119,10 @@ }, "packages/core": { "name": "@datastream/core", - "version": "0.3.0", + "version": "0.3.1", "license": "MIT", "devDependencies": { - "@datastream/object": "0.3.0" + "@datastream/object": "0.3.1" }, "engines": { "node": ">=24" @@ -10130,11 +10130,11 @@ }, "packages/csv": { "name": "@datastream/csv", - "version": "0.3.0", + "version": "0.3.1", "license": "MIT", "dependencies": { - "@datastream/core": "0.3.0", - "@datastream/object": "0.3.0" + "@datastream/core": "0.3.1", + "@datastream/object": "0.3.1" }, "engines": { "node": ">=24" @@ -10142,10 +10142,10 @@ }, "packages/digest": { "name": "@datastream/digest", - "version": "0.3.0", + "version": "0.3.1", "license": "MIT", "dependencies": { - "@datastream/core": "0.3.0", + "@datastream/core": "0.3.1", "hash-wasm": "4.12.0" }, "engines": { @@ -10154,10 +10154,10 @@ }, "packages/encrypt": { "name": "@datastream/encrypt", - "version": "0.3.0", + "version": "0.3.1", "license": "MIT", "dependencies": { - "@datastream/core": "0.3.0" + "@datastream/core": "0.3.1" }, "engines": { "node": ">=24" @@ -10173,10 +10173,10 @@ }, "packages/fetch": { "name": "@datastream/fetch", - "version": "0.3.0", + "version": "0.3.1", "license": "MIT", "dependencies": { - "@datastream/core": "0.3.0" + "@datastream/core": "0.3.1" }, "engines": { "node": ">=24" @@ -10184,10 +10184,10 @@ }, "packages/file": { "name": "@datastream/file", - "version": "0.3.0", + "version": "0.3.1", "license": "MIT", "dependencies": { - "@datastream/core": "0.3.0" + "@datastream/core": "0.3.1" }, "engines": { "node": ">=24" @@ -10195,10 +10195,10 @@ }, "packages/indexeddb": { "name": "@datastream/indexeddb", - "version": "0.3.0", + "version": "0.3.1", "license": "MIT", "dependencies": { - "@datastream/core": "0.3.0", + "@datastream/core": "0.3.1", "idb": "8.0.3" }, "engines": { @@ -10207,10 +10207,10 @@ }, "packages/ipfs": { "name": "@datastream/ipfs", - "version": "0.3.0", + "version": "0.3.1", "license": "MIT", "dependencies": { - "@datastream/core": "0.3.0" + "@datastream/core": "0.3.1" }, "engines": { "node": ">=24" @@ -10218,10 +10218,10 @@ }, "packages/json": { "name": "@datastream/json", - "version": "0.3.0", + "version": "0.3.1", "license": "MIT", "dependencies": { - "@datastream/core": "0.3.0" + "@datastream/core": "0.3.1" }, "engines": { "node": ">=24" @@ -10229,10 +10229,10 @@ }, "packages/object": { "name": "@datastream/object", - "version": "0.3.0", + "version": "0.3.1", "license": "MIT", "dependencies": { - "@datastream/core": "0.3.0" + "@datastream/core": "0.3.1" }, "engines": { "node": ">=24" @@ -10240,10 +10240,10 @@ }, "packages/string": { "name": "@datastream/string", - "version": "0.3.0", + "version": "0.3.1", "license": "MIT", "dependencies": { - "@datastream/core": "0.3.0" + "@datastream/core": "0.3.1" }, "engines": { "node": ">=24" @@ -10251,10 +10251,10 @@ }, "packages/validate": { "name": "@datastream/validate", - "version": "0.3.0", + "version": "0.3.1", "license": "MIT", "dependencies": { - "@datastream/core": "0.3.0", + "@datastream/core": "0.3.1", "ajv-cmd": "0.11.0" }, "engines": { @@ -10262,7 +10262,7 @@ } }, "websites/datastream.js.org": { - "version": "0.3.0", + "version": "0.3.1", "dependencies": { "@plausible-analytics/tracker": "0.4.4", "@willfarrell-ds/svelte": "0.0.0-alpha.6", diff --git a/package.json b/package.json index 3e5bd3e..d172920 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@datastream/monorepo", - "version": "0.3.0", + "version": "0.3.1", "description": "Streams made easy.", "private": true, "type": "module", diff --git a/packages/aws/cloudwatch-logs.d.ts b/packages/aws/cloudwatch-logs.d.ts index 3933bd6..d081df1 100644 --- a/packages/aws/cloudwatch-logs.d.ts +++ b/packages/aws/cloudwatch-logs.d.ts @@ -1,6 +1,6 @@ // Copyright 2026 will Farrell, and datastream contributors. // SPDX-License-Identifier: MIT -import type { StreamOptions } from "@datastream/core"; +import type { DatastreamReadable, StreamOptions } from "@datastream/core"; export function awsCloudWatchLogsSetClient(cwlClient: unknown): void; @@ -18,7 +18,7 @@ export function awsCloudWatchLogsGetLogEventsStream( [key: string]: unknown; }, streamOptions?: StreamOptions, -): Promise; +): Promise; export function awsCloudWatchLogsFilterLogEventsStream( options: { @@ -33,4 +33,4 @@ export function awsCloudWatchLogsFilterLogEventsStream( [key: string]: unknown; }, streamOptions?: StreamOptions, -): Promise; +): Promise; diff --git a/packages/aws/cloudwatch-logs.js b/packages/aws/cloudwatch-logs.js index 9f3a803..90c1bb0 100644 --- a/packages/aws/cloudwatch-logs.js +++ b/packages/aws/cloudwatch-logs.js @@ -14,24 +14,26 @@ export const awsCloudWatchLogsSetClient = (cwlClient) => { export const awsCloudWatchLogsGetLogEventsStream = async ( options, - _streamOptions = {}, + streamOptions = {}, ) => { const { pollingActive, pollingDelay = 1000, ...cwlOptions } = options; cwlOptions.startFromHead ??= true; - async function* command(options) { + async function* command(opts) { let previousToken; let expectMore = true; while (expectMore) { - const response = await client.send(new GetLogEventsCommand(options)); + const response = await client.send(new GetLogEventsCommand(opts), { + abortSignal: streamOptions.signal, + }); const events = response.events ?? []; for (const item of events) { yield item; } const tokenUnchanged = response.nextForwardToken === previousToken || - response.nextForwardToken === options.nextToken; + response.nextForwardToken === opts.nextToken; previousToken = response.nextForwardToken; - options.nextToken = response.nextForwardToken; + opts.nextToken = response.nextForwardToken; if (tokenUnchanged) { if (pollingActive) { @@ -44,25 +46,27 @@ export const awsCloudWatchLogsGetLogEventsStream = async ( } } } - return command(cwlOptions); + return command({ ...cwlOptions }); }; export const awsCloudWatchLogsFilterLogEventsStream = async ( options, - _streamOptions = {}, + streamOptions = {}, ) => { - async function* command(options) { + async function* command(opts) { let expectMore = true; while (expectMore) { - const response = await client.send(new FilterLogEventsCommand(options)); + const response = await client.send(new FilterLogEventsCommand(opts), { + abortSignal: streamOptions.signal, + }); for (const item of response.events ?? []) { yield item; } - options.nextToken = response.nextToken; + opts.nextToken = response.nextToken; expectMore = !!response.nextToken; } } - return command(options); + return command({ ...options }); }; export default { diff --git a/packages/aws/cloudwatch-logs.test.js b/packages/aws/cloudwatch-logs.test.js index 92208fe..365c8a7 100644 --- a/packages/aws/cloudwatch-logs.test.js +++ b/packages/aws/cloudwatch-logs.test.js @@ -201,3 +201,23 @@ test(`${variant}: awsCloudWatchLogsFilterLogEventsStream should handle empty eve deepStrictEqual(output, []); }); + +// *** AbortSignal *** // +test(`${variant}: awsCloudWatchLogsGetLogEventsStream should pass signal to client`, async (_t) => { + const client = mockClient(CloudWatchLogsClient); + awsCloudWatchLogsSetClient(client); + client.on(GetLogEventsCommand).resolves({ + events: [{ message: "log line" }], + nextForwardToken: "same-token", + }); + + const controller = new AbortController(); + const options = { logGroupName: "g", logStreamName: "s" }; + const stream = await awsCloudWatchLogsGetLogEventsStream(options, { + signal: controller.signal, + }); + await streamToArray(stream); + + const calls = client.commandCalls(GetLogEventsCommand); + deepStrictEqual(calls[0].args[1]?.abortSignal, controller.signal); +}); diff --git a/packages/aws/dynamodb.d.ts b/packages/aws/dynamodb.d.ts index 43ed3c8..9e03af1 100644 --- a/packages/aws/dynamodb.d.ts +++ b/packages/aws/dynamodb.d.ts @@ -1,6 +1,10 @@ // Copyright 2026 will Farrell, and datastream contributors. // SPDX-License-Identifier: MIT -import type { StreamOptions } from "@datastream/core"; +import type { + DatastreamReadable, + DatastreamWritable, + StreamOptions, +} from "@datastream/core"; export function awsDynamoDBSetClient( ddbClient: unknown, @@ -14,7 +18,7 @@ export function awsDynamoDBQueryStream( [key: string]: unknown; }, streamOptions?: StreamOptions, -): Promise; +): Promise; export function awsDynamoDBScanStream( options: { @@ -23,7 +27,7 @@ export function awsDynamoDBScanStream( [key: string]: unknown; }, streamOptions?: StreamOptions, -): Promise; +): Promise; export function awsDynamoDBExecuteStatementStream( options: { @@ -33,7 +37,7 @@ export function awsDynamoDBExecuteStatementStream( [key: string]: unknown; }, streamOptions?: StreamOptions, -): Promise; +): Promise; export function awsDynamoDBGetItemStream( options: { @@ -45,7 +49,7 @@ export function awsDynamoDBGetItemStream( [key: string]: unknown; }, streamOptions?: StreamOptions, -): Promise; +): Promise; export function awsDynamoDBPutItemStream( options: { @@ -54,7 +58,7 @@ export function awsDynamoDBPutItemStream( [key: string]: unknown; }, streamOptions?: StreamOptions, -): unknown; +): DatastreamWritable; export function awsDynamoDBDeleteItemStream( options: { @@ -63,4 +67,4 @@ export function awsDynamoDBDeleteItemStream( [key: string]: unknown; }, streamOptions?: StreamOptions, -): unknown; +): DatastreamWritable; diff --git a/packages/aws/dynamodb.js b/packages/aws/dynamodb.js index edf1972..c3360bf 100644 --- a/packages/aws/dynamodb.js +++ b/packages/aws/dynamodb.js @@ -20,57 +20,57 @@ awsDynamoDBSetClient(client); // options = {TableName, ...} export const awsDynamoDBQueryStream = async (options, streamOptions = {}) => { - async function* command(options) { + async function* command(opts) { let expectMore = true; while (expectMore) { - const response = await client.send(new QueryCommand(options), { + const response = await client.send(new QueryCommand(opts), { abortSignal: streamOptions.signal, }); for (const item of response.Items) { yield item; } - options.ExclusiveStartKey = response.LastEvaluatedKey; + opts.ExclusiveStartKey = response.LastEvaluatedKey; expectMore = !!response.LastEvaluatedKey; } } - return command(options); + return command({ ...options }); }; export const awsDynamoDBScanStream = async (options, streamOptions = {}) => { - async function* command(options) { + async function* command(opts) { let expectMore = true; while (expectMore) { - const response = await client.send(new ScanCommand(options), { + const response = await client.send(new ScanCommand(opts), { abortSignal: streamOptions.signal, }); for (const item of response.Items) { yield item; } - options.ExclusiveStartKey = response.LastEvaluatedKey; + opts.ExclusiveStartKey = response.LastEvaluatedKey; expectMore = !!response.LastEvaluatedKey; } } - return command(options); + return command({ ...options }); }; export const awsDynamoDBExecuteStatementStream = async ( options, streamOptions = {}, ) => { - async function* command(options) { + async function* command(opts) { let expectMore = true; while (expectMore) { - const response = await client.send(new ExecuteStatementCommand(options), { + const response = await client.send(new ExecuteStatementCommand(opts), { abortSignal: streamOptions.signal, }); for (const item of response.Items ?? []) { yield item; } - options.NextToken = response.NextToken; + opts.NextToken = response.NextToken; expectMore = !!response.NextToken; } } - return command(options); + return command({ ...options }); }; export const awsDynamoDBGetItemStream = async (options, streamOptions = {}) => { diff --git a/packages/aws/dynamodb.test.js b/packages/aws/dynamodb.test.js index 13cb7dd..cc78a98 100644 --- a/packages/aws/dynamodb.test.js +++ b/packages/aws/dynamodb.test.js @@ -593,6 +593,26 @@ test(`${variant}: awsDynamoDBPutItemStream should pass abort signal to batch wri deepStrictEqual(calls[0].args[1]?.abortSignal, controller.signal); }); +// *** options mutation *** // +test(`${variant}: awsDynamoDBQueryStream should not mutate caller options`, async (_t) => { + const client = mockClient(DynamoDBClient); + awsDynamoDBSetClient(client); + client.on(QueryCommand).resolves({ + Items: [{ key: "a" }], + LastEvaluatedKey: { key: "a" }, + }); + client.on(QueryCommand).resolves({ + Items: [{ key: "b" }], + }); + + const options = { TableName: "T" }; + const optionsCopy = { ...options }; + const stream = await awsDynamoDBQueryStream(options); + await streamToArray(stream); + + deepStrictEqual(options, optionsCopy); +}); + test(`${variant}: default export should include all stream functions`, (_t) => { deepStrictEqual(Object.keys(dynamodbDefault).sort(), [ "deleteItemStream", diff --git a/packages/aws/index.tst.ts b/packages/aws/index.tst.ts index c70c7f2..4c251dd 100644 --- a/packages/aws/index.tst.ts +++ b/packages/aws/index.tst.ts @@ -1,3 +1,5 @@ +/// +/// import { awsCloudWatchLogsFilterLogEventsStream, awsCloudWatchLogsGetLogEventsStream, diff --git a/packages/aws/kinesis.d.ts b/packages/aws/kinesis.d.ts index be8e906..ff6eee2 100644 --- a/packages/aws/kinesis.d.ts +++ b/packages/aws/kinesis.d.ts @@ -1,6 +1,10 @@ // Copyright 2026 will Farrell, and datastream contributors. // SPDX-License-Identifier: MIT -import type { StreamOptions } from "@datastream/core"; +import type { + DatastreamReadable, + DatastreamWritable, + StreamOptions, +} from "@datastream/core"; export function awsKinesisSetClient(kinesisClient: unknown): void; @@ -13,7 +17,7 @@ export function awsKinesisGetRecordsStream( [key: string]: unknown; }, streamOptions?: StreamOptions, -): Promise; +): Promise; export function awsKinesisPutRecordsStream( options: { @@ -23,4 +27,4 @@ export function awsKinesisPutRecordsStream( [key: string]: unknown; }, streamOptions?: StreamOptions, -): unknown; +): DatastreamWritable; diff --git a/packages/aws/kinesis.js b/packages/aws/kinesis.js index f07cc66..aa82d64 100644 --- a/packages/aws/kinesis.js +++ b/packages/aws/kinesis.js @@ -15,25 +15,28 @@ export const awsKinesisSetClient = (kinesisClient) => { export const awsKinesisGetRecordsStream = async ( options, - _streamOptions = {}, + streamOptions = {}, ) => { const { pollingActive, pollingDelay = 1000, ...kinesisOptions } = options; - async function* command(options) { + async function* command(opts) { let expectMore = true; while (expectMore) { - const response = await client.send(new GetRecordsCommand(options)); + const response = await client.send(new GetRecordsCommand(opts), { + abortSignal: streamOptions.signal, + }); const records = response.Records ?? []; for (const item of records) { yield item; } - options.ShardIterator = response.NextShardIterator; - expectMore = pollingActive || records.length > 0; + opts.ShardIterator = response.NextShardIterator; + expectMore = + opts.ShardIterator !== null && (pollingActive || records.length > 0); if (pollingActive && records.length === 0 && pollingDelay > 0) { await new Promise((resolve) => setTimeout(resolve, pollingDelay)); } } } - return command(kinesisOptions); + return command({ ...kinesisOptions }); }; export const awsKinesisPutRecordsStream = (options, streamOptions = {}) => { diff --git a/packages/aws/kinesis.test.js b/packages/aws/kinesis.test.js index 1df3801..1ee45a5 100644 --- a/packages/aws/kinesis.test.js +++ b/packages/aws/kinesis.test.js @@ -188,3 +188,23 @@ test(`${variant}: awsKinesisPutRecordsStream should handle empty input`, async ( deepStrictEqual(result, {}); }); + +// *** AbortSignal *** // +test(`${variant}: awsKinesisGetRecordsStream should pass signal to client`, async (_t) => { + const client = mockClient(KinesisClient); + awsKinesisSetClient(client); + client.on(GetRecordsCommand).resolves({ + Records: [{ Data: "a" }], + NextShardIterator: null, + }); + + const controller = new AbortController(); + const options = { ShardIterator: "iter-1" }; + const stream = await awsKinesisGetRecordsStream(options, { + signal: controller.signal, + }); + await streamToArray(stream); + + const calls = client.commandCalls(GetRecordsCommand); + deepStrictEqual(calls[0].args[1]?.abortSignal, controller.signal); +}); diff --git a/packages/aws/lambda.d.ts b/packages/aws/lambda.d.ts index ef15bb3..0e45de2 100644 --- a/packages/aws/lambda.d.ts +++ b/packages/aws/lambda.d.ts @@ -1,11 +1,11 @@ // Copyright 2026 will Farrell, and datastream contributors. // SPDX-License-Identifier: MIT -import type { StreamOptions } from "@datastream/core"; +import type { DatastreamReadable, StreamOptions } from "@datastream/core"; export function awsLambdaSetClient(lambdaClient: unknown): void; export function awsLambdaReadableStream( lambdaOptions: Record | Record[], streamOptions?: StreamOptions, -): unknown; +): DatastreamReadable; export { awsLambdaReadableStream as awsLambdaResponseStream }; diff --git a/packages/aws/package.json b/packages/aws/package.json index e7b912e..f791f3f 100644 --- a/packages/aws/package.json +++ b/packages/aws/package.json @@ -1,6 +1,6 @@ { "name": "@datastream/aws", - "version": "0.3.0", + "version": "0.3.1", "description": "AWS service streaming integrations for CloudWatch Logs, DynamoDB, Kinesis, Lambda, S3, SNS, and SQS", "type": "module", "engines": { @@ -148,7 +148,7 @@ }, "homepage": "https://datastream.js.org", "dependencies": { - "@datastream/core": "0.3.0" + "@datastream/core": "0.3.1" }, "peerDependencies": { "@aws-sdk/client-cloudwatch-logs": "^3.0.0", diff --git a/packages/aws/s3.d.ts b/packages/aws/s3.d.ts index 0db1863..ae63e83 100644 --- a/packages/aws/s3.d.ts +++ b/packages/aws/s3.d.ts @@ -1,6 +1,12 @@ // Copyright 2026 will Farrell, and datastream contributors. // SPDX-License-Identifier: MIT -import type { StreamOptions, StreamResult } from "@datastream/core"; +import type { + DatastreamPassThrough, + DatastreamReadable, + DatastreamWritable, + StreamOptions, + StreamResult, +} from "@datastream/core"; export function awsS3SetClient(s3Client: unknown): void; @@ -12,7 +18,7 @@ export function awsS3GetObjectStream( [key: string]: unknown; }, streamOptions?: StreamOptions, -): Promise; +): Promise; export function awsS3PutObjectStream( options: { @@ -24,7 +30,7 @@ export function awsS3PutObjectStream( [key: string]: unknown; }, streamOptions?: StreamOptions, -): unknown & { +): DatastreamWritable & { result: () => Promise>; }; @@ -35,7 +41,7 @@ export function awsS3ChecksumStream( resultKey?: string; }, streamOptions?: StreamOptions, -): unknown & { +): DatastreamPassThrough & { result: () => StreamResult<{ checksum: string; checksums: string[]; diff --git a/packages/aws/sns.d.ts b/packages/aws/sns.d.ts index 65fe082..b4f3eb5 100644 --- a/packages/aws/sns.d.ts +++ b/packages/aws/sns.d.ts @@ -1,6 +1,6 @@ // Copyright 2026 will Farrell, and datastream contributors. // SPDX-License-Identifier: MIT -import type { StreamOptions } from "@datastream/core"; +import type { DatastreamWritable, StreamOptions } from "@datastream/core"; export function awsSNSSetClient(snsClient: unknown): void; @@ -11,4 +11,4 @@ export function awsSNSPublishMessageStream( [key: string]: unknown; }, streamOptions?: StreamOptions, -): unknown; +): DatastreamWritable; diff --git a/packages/aws/sqs.d.ts b/packages/aws/sqs.d.ts index 1f47400..f1f3013 100644 --- a/packages/aws/sqs.d.ts +++ b/packages/aws/sqs.d.ts @@ -1,6 +1,10 @@ // Copyright 2026 will Farrell, and datastream contributors. // SPDX-License-Identifier: MIT -import type { StreamOptions } from "@datastream/core"; +import type { + DatastreamReadable, + DatastreamWritable, + StreamOptions, +} from "@datastream/core"; export function awsSQSSetClient(sqsClient: unknown): void; @@ -13,7 +17,7 @@ export function awsSQSReceiveMessageStream( [key: string]: unknown; }, streamOptions?: StreamOptions, -): Promise; +): Promise; export function awsSQSDeleteMessageStream( options: { @@ -22,7 +26,7 @@ export function awsSQSDeleteMessageStream( [key: string]: unknown; }, streamOptions?: StreamOptions, -): unknown; +): DatastreamWritable; export function awsSQSSendMessageStream( options: { @@ -31,4 +35,4 @@ export function awsSQSSendMessageStream( [key: string]: unknown; }, streamOptions?: StreamOptions, -): unknown; +): DatastreamWritable; diff --git a/packages/base64/index.d.ts b/packages/base64/index.d.ts index ecf1045..1c809b3 100644 --- a/packages/base64/index.d.ts +++ b/packages/base64/index.d.ts @@ -1,15 +1,15 @@ // Copyright 2026 will Farrell, and datastream contributors. // SPDX-License-Identifier: MIT -import type { StreamOptions } from "@datastream/core"; +import type { DatastreamTransform, StreamOptions } from "@datastream/core"; export function base64EncodeStream( - options?: Record, + options?: {}, streamOptions?: StreamOptions, -): unknown; +): DatastreamTransform; export function base64DecodeStream( - options?: Record, + options?: {}, streamOptions?: StreamOptions, -): unknown; +): DatastreamTransform; declare const _default: { encodeStream: typeof base64EncodeStream; diff --git a/packages/base64/index.tst.ts b/packages/base64/index.tst.ts index 279bd4c..a69f8e2 100644 --- a/packages/base64/index.tst.ts +++ b/packages/base64/index.tst.ts @@ -1,3 +1,5 @@ +/// +/// import _default, { base64DecodeStream, base64EncodeStream, diff --git a/packages/base64/package.json b/packages/base64/package.json index 5809708..01c6532 100644 --- a/packages/base64/package.json +++ b/packages/base64/package.json @@ -1,6 +1,6 @@ { "name": "@datastream/base64", - "version": "0.3.0", + "version": "0.3.1", "description": "Base64 encoding and decoding transform streams", "type": "module", "engines": { @@ -60,6 +60,6 @@ }, "homepage": "https://datastream.js.org", "dependencies": { - "@datastream/core": "0.3.0" + "@datastream/core": "0.3.1" } } diff --git a/packages/charset/decode.d.ts b/packages/charset/decode.d.ts index be4f526..fc27289 100644 --- a/packages/charset/decode.d.ts +++ b/packages/charset/decode.d.ts @@ -1,8 +1,8 @@ // Copyright 2026 will Farrell, and datastream contributors. // SPDX-License-Identifier: MIT -import type { StreamOptions } from "@datastream/core"; +import type { DatastreamTransform, StreamOptions } from "@datastream/core"; export function charsetDecodeStream( options?: { charset?: string }, streamOptions?: StreamOptions, -): unknown; +): DatastreamTransform; diff --git a/packages/charset/decode.web.js b/packages/charset/decode.web.js index 5e89a13..874e6bb 100644 --- a/packages/charset/decode.web.js +++ b/packages/charset/decode.web.js @@ -2,8 +2,55 @@ // SPDX-License-Identifier: MIT /* global TextDecoderStream */ +const supportedEncodings = new Set([ + "utf-8", + "utf-16le", + "utf-16be", + "ibm866", + "iso-8859-2", + "iso-8859-3", + "iso-8859-4", + "iso-8859-5", + "iso-8859-6", + "iso-8859-7", + "iso-8859-8", + "iso-8859-8-i", + "iso-8859-10", + "iso-8859-13", + "iso-8859-14", + "iso-8859-15", + "iso-8859-16", + "koi8-r", + "koi8-u", + "macintosh", + "windows-874", + "windows-1250", + "windows-1251", + "windows-1252", + "windows-1253", + "windows-1254", + "windows-1255", + "windows-1256", + "windows-1257", + "windows-1258", + "x-mac-cyrillic", + "gbk", + "gb18030", + "big5", + "euc-jp", + "iso-2022-jp", + "shift_jis", + "euc-kr", + "replacement", + "x-user-defined", +]); + export const charsetDecodeStream = ({ charset } = {}, _streamOptions = {}) => { - // doesn't support signal? + if (charset !== null && !supportedEncodings.has(charset.toLowerCase())) { + throw new Error( + `charsetDecodeStream: Unsupported web encoding "${charset}"`, + ); + } return new TextDecoderStream(charset); }; diff --git a/packages/charset/detect.d.ts b/packages/charset/detect.d.ts index 6dd1c61..816d212 100644 --- a/packages/charset/detect.d.ts +++ b/packages/charset/detect.d.ts @@ -1,12 +1,16 @@ // Copyright 2026 will Farrell, and datastream contributors. // SPDX-License-Identifier: MIT -import type { StreamOptions, StreamResult } from "@datastream/core"; +import type { + DatastreamPassThrough, + StreamOptions, + StreamResult, +} from "@datastream/core"; export function charsetDetectStream( options?: { resultKey?: string; }, streamOptions?: StreamOptions, -): unknown & { +): DatastreamPassThrough & { result: () => StreamResult<{ charset: string; confidence: number }>; }; diff --git a/packages/charset/detect.js b/packages/charset/detect.js index 92793f0..6ebc404 100644 --- a/packages/charset/detect.js +++ b/packages/charset/detect.js @@ -35,8 +35,10 @@ const charsetKeys = [ export const charsetDetectStream = ({ resultKey } = {}, streamOptions = {}) => { const charsets = Object.fromEntries(charsetKeys.map((k) => [k, 0])); + let chunkCount = 0; const passThrough = (chunk) => { const matches = detect(chunk); + chunkCount++; if (matches.length) { for (const match of matches) { charsets[match.charsetName] += match.confidence; @@ -45,8 +47,12 @@ export const charsetDetectStream = ({ resultKey } = {}, streamOptions = {}) => { }; const stream = createPassThroughStream(passThrough, streamOptions); stream.result = () => { + const divisor = chunkCount || 1; const values = Object.entries(charsets) - .map(([charset, confidence]) => ({ charset, confidence })) + .map(([charset, confidence]) => ({ + charset, + confidence: confidence / divisor, + })) .sort((a, b) => b.confidence - a.confidence); return { key: resultKey ?? "charset", value: values[0] }; }; diff --git a/packages/charset/encode.d.ts b/packages/charset/encode.d.ts index 60e7fde..2a63803 100644 --- a/packages/charset/encode.d.ts +++ b/packages/charset/encode.d.ts @@ -1,8 +1,8 @@ // Copyright 2026 will Farrell, and datastream contributors. // SPDX-License-Identifier: MIT -import type { StreamOptions } from "@datastream/core"; +import type { DatastreamTransform, StreamOptions } from "@datastream/core"; export function charsetEncodeStream( options?: { charset?: string }, streamOptions?: StreamOptions, -): unknown; +): DatastreamTransform; diff --git a/packages/charset/encode.web.js b/packages/charset/encode.web.js index 22f5a17..21dc5c2 100644 --- a/packages/charset/encode.web.js +++ b/packages/charset/encode.web.js @@ -3,8 +3,12 @@ /* global TextEncoderStream */ export const charsetEncodeStream = ({ charset } = {}, _streamOptions = {}) => { - // doesn't support signal? - return new TextEncoderStream(charset); + if (charset !== null && charset.toUpperCase() !== "UTF-8") { + throw new Error( + `charsetEncodeStream: Web only supports UTF-8 encoding, got "${charset}"`, + ); + } + return new TextEncoderStream(); }; export default charsetEncodeStream; diff --git a/packages/charset/index.test.js b/packages/charset/index.test.js index da29d78..bc3b17a 100644 --- a/packages/charset/index.test.js +++ b/packages/charset/index.test.js @@ -1,4 +1,4 @@ -import { deepStrictEqual, strictEqual } from "node:assert"; +import { deepStrictEqual, ok, strictEqual } from "node:assert"; import test from "node:test"; import { charsetDecodeStream, @@ -332,3 +332,45 @@ test(`${variant}: charsetDetectStream instances should not share state`, async ( strictEqual(typeof result1.value.confidence, "number"); strictEqual(typeof result2.value.confidence, "number"); }); + +// *** charsetDetectStream confidence normalization *** // +test(`${variant}: charsetDetectStream should normalize confidence across chunks`, async (_t) => { + // Send many chunks — confidence should be averaged, not sum to huge numbers + const chunks = Array.from({ length: 100 }, () => Buffer.from("Hello World")); + const streams = [createReadableStream(chunks), charsetDetectStream()]; + + await pipeline(streams); + const { value } = streams[1].result(); + + // Confidence should be a reasonable value (0-100 range), not 100x the single-chunk value + ok( + value.confidence <= 100, + `confidence ${value.confidence} should be <= 100`, + ); +}); + +// *** charsetEncodeStream charset validation (web-only) *** // +// Web implementation only supports UTF-8; node supports all charsets via iconv +if (variant === "webstream") { + test(`${variant}: charsetEncodeStream should throw for non-UTF-8 charset`, { + skip: "requires web implementation", + }, async (_t) => { + try { + charsetEncodeStream({ charset: "iso-8859-1" }); + throw new Error("Expected error"); + } catch (e) { + ok(e.message.includes("UTF-8")); + } + }); + + test(`${variant}: charsetDecodeStream should throw for unsupported charset`, { + skip: "requires web implementation", + }, async (_t) => { + try { + charsetDecodeStream({ charset: "INVALID-CHARSET-999" }); + throw new Error("Expected error"); + } catch (e) { + ok(e.message !== "Expected error"); + } + }); +} diff --git a/packages/charset/index.tst.ts b/packages/charset/index.tst.ts index f6bcf00..e47d6b8 100644 --- a/packages/charset/index.tst.ts +++ b/packages/charset/index.tst.ts @@ -1,3 +1,5 @@ +/// +/// import { charsetDecodeStream, charsetDetectStream, diff --git a/packages/charset/package.json b/packages/charset/package.json index dd13be7..d460327 100644 --- a/packages/charset/package.json +++ b/packages/charset/package.json @@ -1,6 +1,6 @@ { "name": "@datastream/charset", - "version": "0.3.0", + "version": "0.3.1", "description": "Character encoding detection, decoding, and conversion streams", "type": "module", "engines": { @@ -108,7 +108,7 @@ }, "homepage": "https://datastream.js.org", "dependencies": { - "@datastream/core": "0.3.0", + "@datastream/core": "0.3.1", "charset-detector": "0.0.2", "iconv-lite": "0.7.2" } diff --git a/packages/compress/brotli.d.ts b/packages/compress/brotli.d.ts index d323e2b..554d847 100644 --- a/packages/compress/brotli.d.ts +++ b/packages/compress/brotli.d.ts @@ -1,12 +1,21 @@ // Copyright 2026 will Farrell, and datastream contributors. // SPDX-License-Identifier: MIT -import type { StreamOptions } from "@datastream/core"; +import type { DatastreamTransform, StreamOptions } from "@datastream/core"; + +export interface BrotliCompressOptions { + quality?: number; + maxOutputSize?: number; +} + +export interface BrotliDecompressOptions { + maxOutputSize?: number; +} export function brotliCompressStream( - options?: { quality?: number }, + options?: BrotliCompressOptions, streamOptions?: StreamOptions, -): unknown; +): DatastreamTransform; export function brotliDecompressStream( - options?: Record, + options?: BrotliDecompressOptions, streamOptions?: StreamOptions, -): unknown; +): DatastreamTransform; diff --git a/packages/compress/brotli.node.js b/packages/compress/brotli.node.js index 094db80..1f75985 100644 --- a/packages/compress/brotli.node.js +++ b/packages/compress/brotli.node.js @@ -29,6 +29,7 @@ export const brotliDecompressStream = (options = {}, streamOptions = {}) => { if (chunk !== null) { outputSize += chunk.length; if (outputSize > maxOutputSize) { + stream.push = originalPush; stream.destroy( new Error( `Decompression output exceeds maxOutputSize (${maxOutputSize} bytes)`, @@ -39,6 +40,9 @@ export const brotliDecompressStream = (options = {}, streamOptions = {}) => { } return originalPush(chunk); }; + stream.on("close", () => { + stream.push = originalPush; + }); } return stream; }; diff --git a/packages/compress/deflate.d.ts b/packages/compress/deflate.d.ts index e6c69f9..632a10f 100644 --- a/packages/compress/deflate.d.ts +++ b/packages/compress/deflate.d.ts @@ -1,12 +1,22 @@ // Copyright 2026 will Farrell, and datastream contributors. // SPDX-License-Identifier: MIT -import type { StreamOptions } from "@datastream/core"; +import type { DatastreamTransform, StreamOptions } from "@datastream/core"; + +export interface DeflateCompressOptions { + quality?: number; + level?: number; + maxOutputSize?: number; +} + +export interface DeflateDecompressOptions { + maxOutputSize?: number; +} export function deflateCompressStream( - options?: Record, + options?: DeflateCompressOptions, streamOptions?: StreamOptions, -): unknown; +): DatastreamTransform; export function deflateDecompressStream( - options?: Record, + options?: DeflateDecompressOptions, streamOptions?: StreamOptions, -): unknown; +): DatastreamTransform; diff --git a/packages/compress/deflate.node.js b/packages/compress/deflate.node.js index 810cb12..ff80036 100644 --- a/packages/compress/deflate.node.js +++ b/packages/compress/deflate.node.js @@ -4,8 +4,31 @@ import { createDeflate, createInflate } from "node:zlib"; // quality -1 - 9 export const deflateCompressStream = (options = {}, _streamOptions = {}) => { - const { quality, ...rest } = options; - return createDeflate({ ...rest, level: rest.level ?? quality }); + const { quality, maxOutputSize, ...rest } = options; + const stream = createDeflate({ ...rest, level: rest.level ?? quality }); + if (maxOutputSize !== null && maxOutputSize !== undefined) { + let outputSize = 0; + const originalPush = stream.push.bind(stream); + stream.push = (chunk) => { + if (chunk !== null) { + outputSize += chunk.length; + if (outputSize > maxOutputSize) { + stream.push = originalPush; + stream.destroy( + new Error( + `Compression output exceeds maxOutputSize (${maxOutputSize} bytes)`, + ), + ); + return false; + } + } + return originalPush(chunk); + }; + stream.on("close", () => { + stream.push = originalPush; + }); + } + return stream; }; export const deflateDecompressStream = (options = {}, streamOptions = {}) => { const { maxOutputSize } = options; @@ -17,6 +40,7 @@ export const deflateDecompressStream = (options = {}, streamOptions = {}) => { if (chunk !== null) { outputSize += chunk.length; if (outputSize > maxOutputSize) { + stream.push = originalPush; stream.destroy( new Error( `Decompression output exceeds maxOutputSize (${maxOutputSize} bytes)`, @@ -27,6 +51,9 @@ export const deflateDecompressStream = (options = {}, streamOptions = {}) => { } return originalPush(chunk); }; + stream.on("close", () => { + stream.push = originalPush; + }); } return stream; }; diff --git a/packages/compress/deflate.web.js b/packages/compress/deflate.web.js index 1eda13f..fdc9a80 100644 --- a/packages/compress/deflate.web.js +++ b/packages/compress/deflate.web.js @@ -6,8 +6,32 @@ // - not supported on firefox - https://bugzilla.mozilla.org/show_bug.cgi?id=1586639 // - not supported in safari -export const deflateCompressStream = (_options = {}, _streamOptions = {}) => { - return new CompressionStream("deflate"); +export const deflateCompressStream = (options = {}, _streamOptions = {}) => { + const { maxOutputSize } = options; + const compressor = new CompressionStream("deflate"); + if (maxOutputSize != null) { + let outputSize = 0; + const transformer = { + transform(chunk, controller) { + outputSize += chunk.byteLength; + if (outputSize > maxOutputSize) { + controller.error( + new Error( + `Compression output exceeds maxOutputSize (${maxOutputSize} bytes)`, + ), + ); + return; + } + controller.enqueue(chunk); + }, + }; + const limiter = new TransformStream(transformer); + return { + readable: compressor.readable.pipeThrough(limiter), + writable: compressor.writable, + }; + } + return compressor; }; export const deflateDecompressStream = (options = {}, _streamOptions = {}) => { const { maxOutputSize } = options; diff --git a/packages/compress/gzip.d.ts b/packages/compress/gzip.d.ts index 74ebad9..7e9c0ff 100644 --- a/packages/compress/gzip.d.ts +++ b/packages/compress/gzip.d.ts @@ -1,12 +1,21 @@ // Copyright 2026 will Farrell, and datastream contributors. // SPDX-License-Identifier: MIT -import type { StreamOptions } from "@datastream/core"; +import type { DatastreamTransform, StreamOptions } from "@datastream/core"; + +export interface GzipCompressOptions { + quality?: number; + maxOutputSize?: number; +} + +export interface GzipDecompressOptions { + maxOutputSize?: number; +} export function gzipCompressStream( - options?: Record, + options?: GzipCompressOptions, streamOptions?: StreamOptions, -): unknown; +): DatastreamTransform; export function gzipDecompressStream( - options?: Record, + options?: GzipDecompressOptions, streamOptions?: StreamOptions, -): unknown; +): DatastreamTransform; diff --git a/packages/compress/gzip.node.js b/packages/compress/gzip.node.js index 566e57a..c0a9255 100644 --- a/packages/compress/gzip.node.js +++ b/packages/compress/gzip.node.js @@ -3,8 +3,32 @@ import { createGunzip, createGzip } from "node:zlib"; // quality -1 - 9 -export const gzipCompressStream = ({ quality } = {}, streamOptions = {}) => { - return createGzip({ ...streamOptions, level: quality }); +export const gzipCompressStream = (options = {}, streamOptions = {}) => { + const { quality, maxOutputSize } = options; + const stream = createGzip({ ...streamOptions, level: quality }); + if (maxOutputSize !== null && maxOutputSize !== undefined) { + let outputSize = 0; + const originalPush = stream.push.bind(stream); + stream.push = (chunk) => { + if (chunk !== null) { + outputSize += chunk.length; + if (outputSize > maxOutputSize) { + stream.push = originalPush; + stream.destroy( + new Error( + `Compression output exceeds maxOutputSize (${maxOutputSize} bytes)`, + ), + ); + return false; + } + } + return originalPush(chunk); + }; + stream.on("close", () => { + stream.push = originalPush; + }); + } + return stream; }; export const gzipDecompressStream = (options = {}, streamOptions = {}) => { const { maxOutputSize } = options; @@ -16,6 +40,7 @@ export const gzipDecompressStream = (options = {}, streamOptions = {}) => { if (chunk !== null) { outputSize += chunk.length; if (outputSize > maxOutputSize) { + stream.push = originalPush; stream.destroy( new Error( `Decompression output exceeds maxOutputSize (${maxOutputSize} bytes)`, @@ -26,6 +51,9 @@ export const gzipDecompressStream = (options = {}, streamOptions = {}) => { } return originalPush(chunk); }; + stream.on("close", () => { + stream.push = originalPush; + }); } return stream; }; diff --git a/packages/compress/gzip.web.js b/packages/compress/gzip.web.js index e6a465b..7776ee1 100644 --- a/packages/compress/gzip.web.js +++ b/packages/compress/gzip.web.js @@ -6,8 +6,32 @@ // - not supported on firefox - https://bugzilla.mozilla.org/show_bug.cgi?id=1586639 // - not supported in safari -export const gzipCompressStream = (_options = {}, _streamOptions = {}) => { - return new CompressionStream("gzip"); +export const gzipCompressStream = (options = {}, _streamOptions = {}) => { + const { maxOutputSize } = options; + const compressor = new CompressionStream("gzip"); + if (maxOutputSize != null) { + let outputSize = 0; + const transformer = { + transform(chunk, controller) { + outputSize += chunk.byteLength; + if (outputSize > maxOutputSize) { + controller.error( + new Error( + `Compression output exceeds maxOutputSize (${maxOutputSize} bytes)`, + ), + ); + return; + } + controller.enqueue(chunk); + }, + }; + const limiter = new TransformStream(transformer); + return { + readable: compressor.readable.pipeThrough(limiter), + writable: compressor.writable, + }; + } + return compressor; }; export const gzipDecompressStream = (options = {}, _streamOptions = {}) => { const { maxOutputSize } = options; diff --git a/packages/compress/index.test.js b/packages/compress/index.test.js index 466c5c5..3d77e7e 100644 --- a/packages/compress/index.test.js +++ b/packages/compress/index.test.js @@ -155,3 +155,93 @@ if (variant === "node") { strictEqual(output, compressibleBody); }); } + +// *** web variant decompression bomb protection *** // +// *** protobuf *** // +let hasProtobuf = false; +try { + const protobuf = await import("protobufjs"); + hasProtobuf = !!protobuf; +} catch { + // protobufjs not installed +} + +if (hasProtobuf) { + const protobuf = await import("protobufjs"); + const { protobufSerializeStream, protobufDeserializeStream } = await import( + "@datastream/compress/protobuf" + ); + + const TestType = new protobuf.Type("TestMessage") + .add(new protobuf.Field("name", 1, "string")) + .add(new protobuf.Field("value", 2, "int32")); + new protobuf.Root().define("test").add(TestType); + + test(`${variant}: protobuf roundtrip serialize/deserialize`, async (_t) => { + const input = [ + { name: "a", value: 1 }, + { name: "b", value: 2 }, + ]; + const serialize = protobufSerializeStream({ Type: TestType }); + const deserialize = protobufDeserializeStream({ Type: TestType }); + const streams = [createReadableStream(input), serialize, deserialize]; + const output = await streamToString(pipejoin(streams)); + ok(output.includes("a")); + }); +} + +if (variant === "webstream") { + test(`${variant}: gzipDecompressStream should enforce maxOutputSize`, async (_t) => { + const input = gzipSync(compressibleBody); + const streams = [ + createReadableStream(input), + gzipDecompressStream({ maxOutputSize: 100 }), + ]; + try { + await pipeline(streams); + throw new Error("Should have thrown"); + } catch (e) { + ok(e.message.includes("maxOutputSize")); + } + }); + + test(`${variant}: deflateDecompressStream should enforce maxOutputSize`, async (_t) => { + const input = deflateSync(compressibleBody); + const streams = [ + createReadableStream(input), + deflateDecompressStream({ maxOutputSize: 100 }), + ]; + try { + await pipeline(streams); + throw new Error("Should have thrown"); + } catch (e) { + ok(e.message.includes("maxOutputSize")); + } + }); + + test(`${variant}: gzipCompressStream should enforce maxOutputSize`, async (_t) => { + const streams = [ + createReadableStream(compressibleBody), + gzipCompressStream({ maxOutputSize: 10 }), + ]; + try { + await pipeline(streams); + throw new Error("Should have thrown"); + } catch (e) { + ok(e.message.includes("maxOutputSize")); + } + }); + + test(`${variant}: deflateCompressStream should enforce maxOutputSize`, async (_t) => { + const streams = [ + createReadableStream(compressibleBody), + deflateCompressStream({ maxOutputSize: 10 }), + ]; + try { + await pipeline(streams); + throw new Error("Should have thrown"); + } catch (e) { + ok(e.message.includes("maxOutputSize")); + } + }); +} diff --git a/packages/compress/index.tst.ts b/packages/compress/index.tst.ts index eebe39d..0a8cd51 100644 --- a/packages/compress/index.tst.ts +++ b/packages/compress/index.tst.ts @@ -1,3 +1,5 @@ +/// +/// import { brotliCompressStream, brotliDecompressStream, diff --git a/packages/compress/package.json b/packages/compress/package.json index ddd7ace..c496530 100644 --- a/packages/compress/package.json +++ b/packages/compress/package.json @@ -1,6 +1,6 @@ { "name": "@datastream/compress", - "version": "0.3.0", + "version": "0.3.1", "description": "Compression and decompression streams for gzip, deflate, brotli, and zstd", "type": "module", "engines": { @@ -140,7 +140,7 @@ }, "homepage": "https://datastream.js.org", "dependencies": { - "@datastream/core": "0.3.0" + "@datastream/core": "0.3.1" }, "peerDependencies": { "brotli-wasm": "^3.0.0", diff --git a/packages/compress/protobuf.d.ts b/packages/compress/protobuf.d.ts index 42ff134..3f55cbe 100644 --- a/packages/compress/protobuf.d.ts +++ b/packages/compress/protobuf.d.ts @@ -1,6 +1,6 @@ // Copyright 2026 will Farrell, and datastream contributors. // SPDX-License-Identifier: MIT -import type { StreamOptions } from "@datastream/core"; +import type { DatastreamTransform, StreamOptions } from "@datastream/core"; export interface ProtobufType { encode(message: unknown): { finish(): Uint8Array }; @@ -11,8 +11,8 @@ export interface ProtobufType { export function protobufSerializeStream( options?: { Type?: ProtobufType }, streamOptions?: StreamOptions, -): unknown; +): DatastreamTransform; export function protobufDeserializeStream( options?: { Type?: ProtobufType }, streamOptions?: StreamOptions, -): unknown; +): DatastreamTransform; diff --git a/packages/compress/zstd.d.ts b/packages/compress/zstd.d.ts index 1bc770d..834bca1 100644 --- a/packages/compress/zstd.d.ts +++ b/packages/compress/zstd.d.ts @@ -1,12 +1,21 @@ // Copyright 2026 will Farrell, and datastream contributors. // SPDX-License-Identifier: MIT -import type { StreamOptions } from "@datastream/core"; +import type { DatastreamTransform, StreamOptions } from "@datastream/core"; + +export interface ZstdCompressOptions { + quality?: number; + maxOutputSize?: number; +} + +export interface ZstdDecompressOptions { + maxOutputSize?: number; +} export function zstdCompressStream( - options?: Record, + options?: ZstdCompressOptions, streamOptions?: StreamOptions, -): unknown; +): DatastreamTransform; export function zstdDecompressStream( - options?: Record, + options?: ZstdDecompressOptions, streamOptions?: StreamOptions, -): unknown; +): DatastreamTransform; diff --git a/packages/compress/zstd.node.js b/packages/compress/zstd.node.js index 1899c16..782617a 100644 --- a/packages/compress/zstd.node.js +++ b/packages/compress/zstd.node.js @@ -22,6 +22,7 @@ export const zstdDecompressStream = (options = {}, _streamOptions = {}) => { if (chunk !== null) { outputSize += chunk.length; if (outputSize > maxOutputSize) { + stream.push = originalPush; stream.destroy( new Error( `Decompression output exceeds maxOutputSize (${maxOutputSize} bytes)`, @@ -32,6 +33,9 @@ export const zstdDecompressStream = (options = {}, _streamOptions = {}) => { } return originalPush(chunk); }; + stream.on("close", () => { + stream.push = originalPush; + }); } return stream; }; diff --git a/packages/core/index.node.js b/packages/core/index.node.js index 7d57e25..8c5b193 100644 --- a/packages/core/index.node.js +++ b/packages/core/index.node.js @@ -28,7 +28,9 @@ export const pipeline = async (streams, streamOptions = {}) => { export const pipejoin = ( streams, onError = (e) => { - throw e; + process.nextTick(() => { + throw e; + }); }, ) => { const pipeline = streams.reduce((pipeline, stream, idx) => { @@ -62,7 +64,7 @@ export const backpressureGauge = (streams) => { const value = values[i]; metrics[keys[i]] = { timeline: [], total: {} }; let timestamp; - let startTimestamp; + const startTimestamp = Date.now(); value.on("pause", () => { timestamp = Date.now(); // process.hrtime.bigint() }); @@ -71,8 +73,6 @@ export const backpressureGauge = (streams) => { // Number.parseInt( (process.hrtime.bigint() - pauseTimestamp).toString() , 10 ) / 1_000_000 // ms const duration = Date.now() - timestamp; metrics[keys[i]].timeline.push({ timestamp, duration }); - } else { - startTimestamp = Date.now(); } }); value.on("end", () => { @@ -289,7 +289,6 @@ export const makeOptions = ({ export const createReadableStream = (input, streamOptions = {}) => { if (input === undefined) { const maxQueueSize = streamOptions.highWaterMark ?? 1024; - let queueSize = 0; const stream = new Readable({ objectMode: streamOptions.objectMode ?? true, highWaterMark: streamOptions.highWaterMark, @@ -297,12 +296,11 @@ export const createReadableStream = (input, streamOptions = {}) => { }); const nativePush = Readable.prototype.push.bind(stream); stream.push = (chunk) => { - if (chunk !== null && queueSize >= maxQueueSize) { + if (chunk !== null && stream.readableLength >= maxQueueSize) { throw new Error( - `createReadableStream queue size (${queueSize}) exceeds limit (${maxQueueSize})`, + `createReadableStream queue size (${stream.readableLength}) exceeds limit (${maxQueueSize})`, ); } - if (chunk !== null) queueSize++; return nativePush(chunk); }; return stream; @@ -321,8 +319,9 @@ export const createReadableStream = (input, streamOptions = {}) => { }; export const createReadableStreamFromString = (input, streamOptions = {}) => { + const size = streamOptions?.chunkSize ?? 16_384; // 16KB + if (size <= 0) throw new Error("chunkSize must be a positive number"); function* iterator(input) { - const size = streamOptions?.chunkSize ?? 16_384; // 16KB let position = 0; const length = input.length; while (position < length) { @@ -337,8 +336,9 @@ export const createReadableStreamFromArrayBuffer = ( input, streamOptions = {}, ) => { + const size = streamOptions?.chunkSize ?? 16_384; // 16KB + if (size <= 0) throw new Error("chunkSize must be a positive number"); function* iterator(input) { - const size = streamOptions?.chunkSize ?? 16_384; // 16KB const bytes = new Uint8Array(input); let position = 0; const length = bytes.byteLength; @@ -483,13 +483,18 @@ export const timeout = (ms, { signal } = {}) => { ); } return new Promise((resolve, reject) => { + let settled = false; const abortHandler = () => { + if (settled) return; + settled = true; clearTimeout(timerId); signal.removeEventListener("abort", abortHandler); reject(new Error("Aborted", { cause: { code: "AbortError" } })); }; if (signal) signal.addEventListener("abort", abortHandler); const timerId = setTimeout(() => { + if (settled) return; + settled = true; if (signal) signal.removeEventListener("abort", abortHandler); resolve(); }, ms); diff --git a/packages/core/index.web.js b/packages/core/index.web.js index b7650ef..f7bf7c2 100644 --- a/packages/core/index.web.js +++ b/packages/core/index.web.js @@ -104,14 +104,15 @@ export const makeOptions = ({ signal, ...streamOptions } = {}) => { + const size = chunkSize != null ? () => chunkSize : undefined; return { writableStrategy: { highWaterMark, - size: { chunk: chunkSize }, + size, }, readableStrategy: { highWaterMark, - size: { chunk: chunkSize }, + size, }, signal, ...streamOptions, @@ -120,6 +121,8 @@ export const makeOptions = ({ export const createReadableStream = (input, streamOptions = {}) => { const maxQueueSize = streamOptions.highWaterMark ?? 1024; + const chunkSize = streamOptions?.chunkSize ?? 16_384; // 16KB + if (chunkSize <= 0) throw new Error("chunkSize must be a positive number"); const queued = []; const { readableStrategy } = makeOptions(streamOptions); const stream = new ReadableStream( @@ -130,7 +133,6 @@ export const createReadableStream = (input, streamOptions = {}) => { controller.enqueue(chunk); } if (typeof input === "string") { - const chunkSize = streamOptions?.chunkSize ?? 16_384; // 16KB let position = 0; const length = input.length; while (position < length) { @@ -146,7 +148,6 @@ export const createReadableStream = (input, streamOptions = {}) => { controller.close(); } else if (typeof input === "object" && input.byteLength) { const bytes = new Uint8Array(input.buffer ?? input); - const chunkSize = streamOptions?.chunkSize ?? 16_384; // 16KB let position = 0; const length = bytes.byteLength; while (position < length) { @@ -299,13 +300,18 @@ export const timeout = (ms, { signal } = {}) => { ); } return new Promise((resolve, reject) => { + let settled = false; const abortHandler = () => { + if (settled) return; + settled = true; clearTimeout(timerId); signal.removeEventListener("abort", abortHandler); reject(new Error("Aborted", { cause: { code: "AbortError" } })); }; if (signal) signal.addEventListener("abort", abortHandler); const timerId = setTimeout(() => { + if (settled) return; + settled = true; if (signal) signal.removeEventListener("abort", abortHandler); resolve(); }, ms); diff --git a/packages/core/package.json b/packages/core/package.json index 1c727fc..c589345 100644 --- a/packages/core/package.json +++ b/packages/core/package.json @@ -1,6 +1,6 @@ { "name": "@datastream/core", - "version": "0.3.0", + "version": "0.3.1", "description": "Stream creation utilities and pipeline functions for Web Streams API and Node.js streams", "type": "module", "engines": { @@ -61,6 +61,6 @@ "homepage": "https://datastream.js.org", "dependencies": {}, "devDependencies": { - "@datastream/object": "0.3.0" + "@datastream/object": "0.3.1" } } diff --git a/packages/csv/index.d.ts b/packages/csv/index.d.ts index 1576c3d..d2a79a2 100644 --- a/packages/csv/index.d.ts +++ b/packages/csv/index.d.ts @@ -1,6 +1,11 @@ // Copyright 2026 will Farrell, and datastream contributors. // SPDX-License-Identifier: MIT -import type { StreamOptions, StreamResult } from "@datastream/core"; +import type { + DatastreamPassThrough, + DatastreamTransform, + StreamOptions, + StreamResult, +} from "@datastream/core"; export interface CsvDelimiters { delimiterChar?: string; @@ -45,7 +50,7 @@ export function csvDetectDelimitersStream( resultKey?: string; }, streamOptions?: StreamOptions, -): unknown & { +): DatastreamPassThrough & { result: () => StreamResult; }; @@ -64,7 +69,7 @@ export function csvDetectHeaderStream( resultKey?: string; }, streamOptions?: StreamOptions, -): unknown & { +): DatastreamPassThrough & { result: () => StreamResult<{ header: string[] }>; }; @@ -95,7 +100,7 @@ export function csvParseStream( escapeChar?: string | (() => string); }, streamOptions?: StreamOptions, -): unknown & { +): DatastreamTransform & { result: () => StreamResult>; }; @@ -106,7 +111,7 @@ export function csvRemoveMalformedRowsStream( resultKey?: string; }, streamOptions?: StreamOptions, -): unknown & { +): DatastreamTransform & { result: () => StreamResult>; }; @@ -116,7 +121,7 @@ export function csvRemoveEmptyRowsStream( resultKey?: string; }, streamOptions?: StreamOptions, -): unknown & { +): DatastreamTransform & { result: () => StreamResult>; }; @@ -128,7 +133,7 @@ export function csvCoerceValuesStream( resultKey?: string; }, streamOptions?: StreamOptions, -): unknown & { +): DatastreamTransform & { result: () => StreamResult>; }; @@ -137,23 +142,23 @@ export function csvInjectHeaderStream( header: string[]; }, streamOptions?: StreamOptions, -): unknown; +): DatastreamTransform; export function csvFormatStream( options?: CsvDelimiters, streamOptions?: StreamOptions, -): unknown; +): DatastreamTransform; export function csvArrayToObject( options: { headers: string[] | (() => string[]); }, streamOptions?: StreamOptions, -): unknown; +): DatastreamTransform; export function csvObjectToArray( options: { headers: string[] | (() => string[]); }, streamOptions?: StreamOptions, -): unknown; +): DatastreamTransform; diff --git a/packages/csv/index.js b/packages/csv/index.js index 310808d..29cbdd4 100644 --- a/packages/csv/index.js +++ b/packages/csv/index.js @@ -917,11 +917,15 @@ const coerceToType = (val, type) => { return Number.isNaN(n) ? val : n; } case "boolean": - return val.toLowerCase() === "true"; + return typeof val === "string" + ? val.toLowerCase() === "true" + : Boolean(val); case "null": return null; - case "date": - return new Date(val); + case "date": { + const d = new Date(val); + return Number.isNaN(d.getTime()) ? val : d; + } case "json": try { return JSON.parse(val); diff --git a/packages/csv/index.tst.ts b/packages/csv/index.tst.ts index abb4fb0..0f82f8c 100644 --- a/packages/csv/index.tst.ts +++ b/packages/csv/index.tst.ts @@ -1,3 +1,5 @@ +/// +/// import type { CsvCoerceType, CsvDelimiters, diff --git a/packages/csv/package.json b/packages/csv/package.json index 1cb73e7..9eb4589 100644 --- a/packages/csv/package.json +++ b/packages/csv/package.json @@ -1,6 +1,6 @@ { "name": "@datastream/csv", - "version": "0.3.0", + "version": "0.3.1", "description": "CSV parsing and formatting transform streams", "type": "module", "engines": { @@ -63,7 +63,7 @@ }, "homepage": "https://datastream.js.org", "dependencies": { - "@datastream/core": "0.3.0", - "@datastream/object": "0.3.0" + "@datastream/core": "0.3.1", + "@datastream/object": "0.3.1" } } diff --git a/packages/digest/index.d.ts b/packages/digest/index.d.ts index 2afcfc7..014bca4 100644 --- a/packages/digest/index.d.ts +++ b/packages/digest/index.d.ts @@ -1,6 +1,10 @@ // Copyright 2026 will Farrell, and datastream contributors. // SPDX-License-Identifier: MIT -import type { StreamOptions, StreamResult } from "@datastream/core"; +import type { + DatastreamPassThrough, + StreamOptions, + StreamResult, +} from "@datastream/core"; export type DigestAlgorithm = | "SHA2-256" @@ -10,7 +14,7 @@ export type DigestAlgorithm = | "SHA3-384" | "SHA3-512"; -type DigestStreamResult = unknown & { +type DigestStreamResult = DatastreamPassThrough & { result: () => StreamResult; }; diff --git a/packages/digest/index.test.js b/packages/digest/index.test.js index 4ee5c48..22db559 100644 --- a/packages/digest/index.test.js +++ b/packages/digest/index.test.js @@ -74,6 +74,52 @@ test(`${variant}: digestStream should use custom resultKey`, async (_t) => { strictEqual(typeof result.checksum, "string"); }); +// *** algorithm variants *** // +test(`${variant}: digestStream should calculate SHA2-384`, async (_t) => { + const streams = [ + createReadableStream("test"), + await digestStream({ algorithm: "SHA2-384" }), + ]; + const result = await pipeline(streams); + strictEqual(result.digest.startsWith("SHA2-384:"), true); +}); + +test(`${variant}: digestStream should calculate SHA2-512`, async (_t) => { + const streams = [ + createReadableStream("test"), + await digestStream({ algorithm: "SHA2-512" }), + ]; + const result = await pipeline(streams); + strictEqual(result.digest.startsWith("SHA2-512:"), true); +}); + +test(`${variant}: digestStream should calculate SHA3-256`, async (_t) => { + const streams = [ + createReadableStream("test"), + await digestStream({ algorithm: "SHA3-256" }), + ]; + const result = await pipeline(streams); + strictEqual(result.digest.startsWith("SHA3-256:"), true); +}); + +test(`${variant}: digestStream should calculate SHA3-384`, async (_t) => { + const streams = [ + createReadableStream("test"), + await digestStream({ algorithm: "SHA3-384" }), + ]; + const result = await pipeline(streams); + strictEqual(result.digest.startsWith("SHA3-384:"), true); +}); + +test(`${variant}: digestStream should calculate SHA3-512`, async (_t) => { + const streams = [ + createReadableStream("test"), + await digestStream({ algorithm: "SHA3-512" }), + ]; + const result = await pipeline(streams); + strictEqual(result.digest.startsWith("SHA3-512:"), true); +}); + // *** default export *** // test(`${variant}: default export should be digestStream`, (_t) => { strictEqual(digestDefault, digestStream); diff --git a/packages/digest/index.tst.ts b/packages/digest/index.tst.ts index 102119e..c730ac7 100644 --- a/packages/digest/index.tst.ts +++ b/packages/digest/index.tst.ts @@ -1,3 +1,5 @@ +/// +/// import type { DigestAlgorithm } from "@datastream/digest"; import { digestStream } from "@datastream/digest"; import { describe, expect, test } from "tstyche"; diff --git a/packages/digest/index.web.js b/packages/digest/index.web.js index a0a6c52..1d2dc51 100644 --- a/packages/digest/index.web.js +++ b/packages/digest/index.web.js @@ -21,6 +21,9 @@ export const digestStream = async ( { algorithm, resultKey }, streamOptions = {}, ) => { + if (!algorithms[algorithm]) { + throw new Error(`Unsupported algorithm: ${algorithm}`); + } const hash = await algorithms[algorithm](); const passThrough = (chunk) => { hash.update(chunk); diff --git a/packages/digest/package.json b/packages/digest/package.json index 619a1f6..d67fe18 100644 --- a/packages/digest/package.json +++ b/packages/digest/package.json @@ -1,6 +1,6 @@ { "name": "@datastream/digest", - "version": "0.3.0", + "version": "0.3.1", "description": "Cryptographic hash digest pass-through streams", "type": "module", "engines": { @@ -60,7 +60,7 @@ }, "homepage": "https://datastream.js.org", "dependencies": { - "@datastream/core": "0.3.0", + "@datastream/core": "0.3.1", "hash-wasm": "4.12.0" } } diff --git a/packages/encrypt/index.d.ts b/packages/encrypt/index.d.ts index e3edcbc..766347a 100644 --- a/packages/encrypt/index.d.ts +++ b/packages/encrypt/index.d.ts @@ -1,13 +1,17 @@ // Copyright 2026 will Farrell, and datastream contributors. // SPDX-License-Identifier: MIT -import type { StreamOptions, StreamResult } from "@datastream/core"; +import type { + DatastreamTransform, + StreamOptions, + StreamResult, +} from "@datastream/core"; export type EncryptAlgorithm = | "AES-256-GCM" | "AES-256-CTR" | "CHACHA20-POLY1305"; -type EncryptStreamResult = unknown & { +type EncryptStreamResult = DatastreamTransform & { result: () => StreamResult<{ algorithm: EncryptAlgorithm; iv: Uint8Array; @@ -36,7 +40,7 @@ export function decryptStream( maxOutputSize?: number; }, streamOptions?: StreamOptions, -): unknown | Promise; +): DatastreamTransform | Promise; export function generateEncryptionKey(options?: { bits?: 128 | 256; diff --git a/packages/encrypt/index.node.js b/packages/encrypt/index.node.js index 141509f..d2a0a0b 100644 --- a/packages/encrypt/index.node.js +++ b/packages/encrypt/index.node.js @@ -41,7 +41,7 @@ const validateAad = (aad) => { }; export const encryptStream = ( - { algorithm = "AES-256-GCM", key, iv, aad } = {}, + { algorithm = "AES-256-GCM", key, iv, aad, maxInputSize } = {}, streamOptions = {}, ) => { const config = algorithmMap[algorithm]; @@ -61,6 +61,22 @@ export const encryptStream = ( if (aad && authAlgorithms.includes(algorithm)) { stream.setAAD(aad); } + if (maxInputSize !== null && maxInputSize !== undefined) { + let inputSize = 0; + const originalWrite = stream._transform.bind(stream); + stream._transform = (chunk, encoding, callback) => { + inputSize += chunk.length; + if (inputSize > maxInputSize) { + callback( + new Error( + `Encryption input exceeds maxInputSize (${maxInputSize} bytes). Use AES-256-CTR for large data.`, + ), + ); + return; + } + originalWrite(chunk, encoding, callback); + }; + } stream.result = () => ({ key: "encrypt", value: { @@ -107,6 +123,7 @@ export const decryptStream = ( if (chunk !== null) { outputSize += chunk.length; if (outputSize > maxOutputSize) { + stream.push = originalPush; stream.destroy( new Error( `Decryption output exceeds maxOutputSize (${maxOutputSize} bytes)`, @@ -117,6 +134,9 @@ export const decryptStream = ( } return originalPush(chunk); }; + stream.on("close", () => { + stream.push = originalPush; + }); } return stream; }; diff --git a/packages/encrypt/index.test.js b/packages/encrypt/index.test.js index 9271546..36fe1a5 100644 --- a/packages/encrypt/index.test.js +++ b/packages/encrypt/index.test.js @@ -273,6 +273,40 @@ test(`${variant}: encryptStream should handle empty input`, async (_t) => { strictEqual(decrypted, ""); }); +// *** maxInputSize *** // +test(`${variant}: encryptStream AES-256-GCM should enforce maxInputSize`, async (_t) => { + const input = "a".repeat(200); + const enc = await encryptStream({ key, maxInputSize: 100 }); + try { + const streams = [createReadableStream(input), enc]; + await pipeline(streams); + throw new Error("Expected maxInputSize error"); + } catch (e) { + strictEqual(e.message.includes("maxInputSize"), true); + } +}); + +test(`${variant}: encryptStream CHACHA20-POLY1305 should enforce maxInputSize`, async (_t) => { + const input = "a".repeat(200); + const enc = await encryptStream({ + key, + algorithm: "CHACHA20-POLY1305", + maxInputSize: 100, + }); + try { + const streams = [createReadableStream(input), enc]; + await pipeline(streams); + throw new Error("Expected maxInputSize error"); + } catch (e) { + strictEqual(e.message.includes("maxInputSize"), true); + } +}); + +// *** generateEncryptionKey error cases *** // +test(`${variant}: generateEncryptionKey should throw for unsupported bits`, (_t) => { + throws(() => generateEncryptionKey({ bits: 512 }), /Unsupported key size/); +}); + // *** default export *** // test(`${variant}: default export should include all functions`, (_t) => { deepStrictEqual(Object.keys(encryptDefault).sort(), [ diff --git a/packages/encrypt/index.tst.ts b/packages/encrypt/index.tst.ts new file mode 100644 index 0000000..dbe6d36 --- /dev/null +++ b/packages/encrypt/index.tst.ts @@ -0,0 +1,73 @@ +/// +/// +import _default, { + decryptStream, + encryptStream, + generateEncryptionKey, +} from "@datastream/encrypt"; +import { describe, expect, test } from "tstyche"; + +describe("encryptStream", () => { + test("returns a stream or promise", () => { + const key = new Uint8Array(32); + expect(encryptStream({ key })).type.not.toBeAssignableTo(); + }); + + test("accepts algorithm option", () => { + const key = new Uint8Array(32); + expect( + encryptStream({ key, algorithm: "AES-256-CTR" }), + ).type.not.toBeAssignableTo(); + }); + + test("accepts maxInputSize option", () => { + const key = new Uint8Array(32); + expect( + encryptStream({ key, maxInputSize: 1024 }), + ).type.not.toBeAssignableTo(); + }); +}); + +describe("decryptStream", () => { + test("returns a stream or promise", () => { + const key = new Uint8Array(32); + const iv = new Uint8Array(12); + expect(decryptStream({ key, iv })).type.not.toBeAssignableTo(); + }); + + test("accepts maxOutputSize option", () => { + const key = new Uint8Array(32); + const iv = new Uint8Array(12); + expect( + decryptStream({ key, iv, maxOutputSize: 1024 }), + ).type.not.toBeAssignableTo(); + }); +}); + +describe("generateEncryptionKey", () => { + test("returns Uint8Array", () => { + expect(generateEncryptionKey()).type.toBeAssignableTo(); + }); + + test("accepts bits option", () => { + expect( + generateEncryptionKey({ bits: 128 }), + ).type.toBeAssignableTo(); + }); +}); + +describe("default export", () => { + test("has encryptStream", () => { + expect(_default.encryptStream).type.toBe(); + }); + + test("has decryptStream", () => { + expect(_default.decryptStream).type.toBe(); + }); + + test("has generateEncryptionKey", () => { + expect(_default.generateEncryptionKey).type.toBe< + typeof generateEncryptionKey + >(); + }); +}); diff --git a/packages/encrypt/index.web.js b/packages/encrypt/index.web.js index 9a05f15..7f1acb4 100644 --- a/packages/encrypt/index.web.js +++ b/packages/encrypt/index.web.js @@ -244,7 +244,10 @@ const aesCtrDecrypt = async ({ key, iv, maxOutputSize }, streamOptions) => { }; // ChaCha20-Poly1305: requires optional peer dep -const chacha20Encrypt = async ({ key, iv, aad }, streamOptions) => { +const chacha20Encrypt = async ( + { key, iv, aad, maxInputSize }, + streamOptions, +) => { validateKey(key); validateAad(aad); let sodium; @@ -258,11 +261,19 @@ const chacha20Encrypt = async ({ key, iv, aad }, streamOptions) => { } iv ??= sodium.randombytes_buf(12); validateIv(iv, "CHACHA20-POLY1305"); + maxInputSize ??= DEFAULT_MAX_INPUT_SIZE; const chunks = []; + let inputSize = 0; let authTag; const transform = (chunk) => { const buf = chunk instanceof Uint8Array ? chunk : new TextEncoder().encode(chunk); + inputSize += buf.byteLength; + if (inputSize > maxInputSize) { + throw new Error( + `Encryption input exceeds maxInputSize (${maxInputSize} bytes). Use AES-256-CTR for large data.`, + ); + } chunks.push(buf); }; const flush = (enqueue) => { @@ -343,7 +354,7 @@ export const encryptStream = async ( return aesCtrEncrypt({ key, iv }, streamOptions); } if (algorithm === "CHACHA20-POLY1305") { - return chacha20Encrypt({ key, iv, aad }, streamOptions); + return chacha20Encrypt({ key, iv, aad, maxInputSize }, streamOptions); } throw new Error(`Unsupported algorithm: ${algorithm}`); }; diff --git a/packages/encrypt/package.json b/packages/encrypt/package.json index d602a36..4b076ef 100644 --- a/packages/encrypt/package.json +++ b/packages/encrypt/package.json @@ -1,6 +1,6 @@ { "name": "@datastream/encrypt", - "version": "0.3.0", + "version": "0.3.1", "description": "Symmetric encryption/decryption streams", "type": "module", "engines": { @@ -60,7 +60,7 @@ }, "homepage": "https://datastream.js.org", "dependencies": { - "@datastream/core": "0.3.0" + "@datastream/core": "0.3.1" }, "peerDependencies": { "libsodium-wrappers": ">=0.7.0" diff --git a/packages/fetch/index.d.ts b/packages/fetch/index.d.ts index d80070e..b54e4ab 100644 --- a/packages/fetch/index.d.ts +++ b/packages/fetch/index.d.ts @@ -1,6 +1,11 @@ // Copyright 2026 will Farrell, and datastream contributors. // SPDX-License-Identifier: MIT -import type { StreamOptions, StreamResult } from "@datastream/core"; +import type { + DatastreamReadable, + DatastreamWritable, + StreamOptions, + StreamResult, +} from "@datastream/core"; export interface FetchOptions { url?: string; @@ -31,7 +36,7 @@ export function fetchWritableStream( options: FetchOptions, streamOptions?: StreamOptions, ): Promise< - unknown & { + DatastreamWritable & { result: () => StreamResult; } >; @@ -40,7 +45,7 @@ export { fetchWritableStream as fetchRequestStream }; export function fetchReadableStream( fetchOptions: FetchOptions | FetchOptions[], streamOptions?: StreamOptions, -): unknown; +): DatastreamReadable; export { fetchReadableStream as fetchResponseStream }; export function fetchRateLimit( diff --git a/packages/fetch/index.js b/packages/fetch/index.js index 9d9491c..e091cd4 100644 --- a/packages/fetch/index.js +++ b/packages/fetch/index.js @@ -22,6 +22,18 @@ const validatePaginationUrl = (nextUrl, origin) => { } }; +const redactUrl = (urlString) => { + try { + const url = new URL(urlString); + if (url.search) url.search = "?[REDACTED]"; + if (url.username) url.username = "[REDACTED]"; + if (url.password) url.password = "[REDACTED]"; + return url.toString(); + } catch { + return "[INVALID URL]"; + } +}; + let defaults = { // custom rateLimit: 0.01, // 100 per sec @@ -100,8 +112,13 @@ async function* fetchGenerator(fetchOptions, streamOptions) { } options.__origin = new URL(options.url).origin; const response = await fetchUnknown(options, streamOptions); - for await (const chunk of response) { - yield chunk; + try { + for await (const chunk of response) { + yield chunk; + } + } catch (error) { + await response?.cancel?.(); + throw error; } // ensure there is rate limiting between req with different options rateLimitTimestamp = options.rateLimitTimestamp; @@ -209,6 +226,7 @@ export const fetchRateLimit = async (options, streamOptions = {}) => { }; const response = await fetch(options.url, fetchInit); if (!response.ok) { + const safeUrl = redactUrl(options.url); // 429 Too Many Requests if (response.status === 429) { options.retryCount = (options.retryCount ?? 0) + 1; @@ -216,7 +234,7 @@ export const fetchRateLimit = async (options, streamOptions = {}) => { if (options.retryCount >= retryMaxCount) { await response.body?.cancel(); throw new Error( - `fetch ${response.status} ${options.method} ${options.url} max retries (${retryMaxCount}) exceeded`, + `fetch ${response.status} ${options.method} ${safeUrl} max retries (${retryMaxCount}) exceeded`, { cause: { status: response.status, @@ -235,16 +253,13 @@ export const fetchRateLimit = async (options, streamOptions = {}) => { return fetchRateLimit(options, streamOptions); } await response.body?.cancel(); - throw new Error( - `fetch ${response.status} ${options.method} ${options.url}`, - { - cause: { - status: response.status, - url: options.url, - method: options.method, - }, + throw new Error(`fetch ${response.status} ${options.method} ${safeUrl}`, { + cause: { + status: response.status, + url: options.url, + method: options.method, }, - ); + }); } return response; }; diff --git a/packages/fetch/index.tst.ts b/packages/fetch/index.tst.ts index 49e8deb..e3904e3 100644 --- a/packages/fetch/index.tst.ts +++ b/packages/fetch/index.tst.ts @@ -1,3 +1,5 @@ +/// +/// import type { FetchOptions } from "@datastream/fetch"; import { fetchRateLimit, diff --git a/packages/fetch/package.json b/packages/fetch/package.json index dbe517c..990837e 100644 --- a/packages/fetch/package.json +++ b/packages/fetch/package.json @@ -1,6 +1,6 @@ { "name": "@datastream/fetch", - "version": "0.3.0", + "version": "0.3.1", "description": "HTTP fetch-based readable and writable streams with pagination and rate limiting", "type": "module", "engines": { @@ -60,6 +60,6 @@ }, "homepage": "https://datastream.js.org", "dependencies": { - "@datastream/core": "0.3.0" + "@datastream/core": "0.3.1" } } diff --git a/packages/file/index.d.ts b/packages/file/index.d.ts index 351804f..15befee 100644 --- a/packages/file/index.d.ts +++ b/packages/file/index.d.ts @@ -1,6 +1,10 @@ // Copyright 2026 will Farrell, and datastream contributors. // SPDX-License-Identifier: MIT -import type { StreamOptions } from "@datastream/core"; +import type { + DatastreamReadable, + DatastreamWritable, + StreamOptions, +} from "@datastream/core"; export interface FilePickerTypes { description?: string; @@ -12,7 +16,7 @@ export function fileReadStream( types?: FilePickerTypes[]; }, streamOptions?: StreamOptions, -): Promise; +): Promise; export function fileWriteStream( options: { @@ -20,7 +24,7 @@ export function fileWriteStream( types?: FilePickerTypes[]; }, streamOptions?: StreamOptions, -): Promise; +): Promise; declare const _default: { readStream: typeof fileReadStream; diff --git a/packages/file/package.json b/packages/file/package.json index 06a04fc..af5f8c8 100644 --- a/packages/file/package.json +++ b/packages/file/package.json @@ -1,6 +1,6 @@ { "name": "@datastream/file", - "version": "0.3.0", + "version": "0.3.1", "description": "File system readable and writable streams with extension type enforcement", "type": "module", "engines": { @@ -60,6 +60,6 @@ }, "homepage": "https://datastream.js.org", "dependencies": { - "@datastream/core": "0.3.0" + "@datastream/core": "0.3.1" } } diff --git a/packages/indexeddb/index.d.ts b/packages/indexeddb/index.d.ts index 5d135be..ee0cc46 100644 --- a/packages/indexeddb/index.d.ts +++ b/packages/indexeddb/index.d.ts @@ -1,6 +1,10 @@ // Copyright 2026 will Farrell, and datastream contributors. // SPDX-License-Identifier: MIT -import type { StreamOptions } from "@datastream/core"; +import type { + DatastreamReadable, + DatastreamWritable, + StreamOptions, +} from "@datastream/core"; export { openDB as indexedDBConnect } from "idb"; @@ -12,7 +16,7 @@ export function indexedDBReadStream( key?: IDBKeyRange | IDBValidKey; }, streamOptions?: StreamOptions, -): Promise; +): Promise; export function indexedDBWriteStream( options: { @@ -20,4 +24,4 @@ export function indexedDBWriteStream( store: string; }, streamOptions?: StreamOptions, -): Promise; +): Promise; diff --git a/packages/indexeddb/index.test.js b/packages/indexeddb/index.test.js index 591e2d7..f281b81 100644 --- a/packages/indexeddb/index.test.js +++ b/packages/indexeddb/index.test.js @@ -112,3 +112,43 @@ if (!isBrowser) { } }); } + +// *** web variant: indexedDBReadStream with index *** // +if (variant === "webstream") { + test(`${variant}: indexedDBReadStream should use index and key when provided`, { + skip: "requires web implementation", + }, async (_t) => { + const mockCursor = { + async *[Symbol.asyncIterator]() { + yield { id: 1, name: "a" }; + yield { id: 2, name: "b" }; + }, + }; + const mockIndex = { + iterate: (_key) => mockCursor, + }; + const mockStore = { + index: (_name) => mockIndex, + [Symbol.asyncIterator]: async function* () { + yield { id: 1, name: "a" }; + yield { id: 2, name: "b" }; + yield { id: 3, name: "c" }; + }, + }; + const mockDb = { + transaction: (_store) => ({ store: mockStore }), + }; + + const stream = await indexedDBReadStream({ + db: mockDb, + store: "test", + index: "name", + key: "a", + }); + const { streamToArray } = await import("@datastream/core"); + const output = await streamToArray(stream); + + // Should return filtered results (2 items from index), not all 3 from store + strictEqual(output.length, 2); + }); +} diff --git a/packages/indexeddb/index.tst.ts b/packages/indexeddb/index.tst.ts new file mode 100644 index 0000000..3187b0f --- /dev/null +++ b/packages/indexeddb/index.tst.ts @@ -0,0 +1,37 @@ +import { + indexedDBConnect, + indexedDBReadStream, + indexedDBWriteStream, +} from "@datastream/indexeddb"; +import { describe, expect, test } from "tstyche"; + +describe("indexedDBConnect", () => { + test("is a function", () => { + expect(indexedDBConnect).type.not.toBeAssignableTo(); + }); +}); + +describe("indexedDBReadStream", () => { + test("accepts options and returns a promise", () => { + const db = {} as unknown; + expect(indexedDBReadStream({ db, store: "test" })).type.toBeAssignableTo< + Promise + >(); + }); + + test("accepts index option", () => { + const db = {} as unknown; + expect( + indexedDBReadStream({ db, store: "test", index: "idx" }), + ).type.not.toBeAssignableTo(); + }); +}); + +describe("indexedDBWriteStream", () => { + test("accepts options and returns a promise", () => { + const db = {} as unknown; + expect(indexedDBWriteStream({ db, store: "test" })).type.toBeAssignableTo< + Promise + >(); + }); +}); diff --git a/packages/indexeddb/index.web.js b/packages/indexeddb/index.web.js index d5a2b62..33c813e 100644 --- a/packages/indexeddb/index.web.js +++ b/packages/indexeddb/index.web.js @@ -9,9 +9,9 @@ export const indexedDBReadStream = async ( { db, store, index, key }, streamOptions = {}, ) => { - const input = db.transaction(store).store; + let input = db.transaction(store).store; if (index && key) { - input.index(index).iterate(key); + input = input.index(index).iterate(key); } return createReadableStream(input, streamOptions); }; diff --git a/packages/indexeddb/package.json b/packages/indexeddb/package.json index d32480c..4f3e53d 100644 --- a/packages/indexeddb/package.json +++ b/packages/indexeddb/package.json @@ -1,6 +1,6 @@ { "name": "@datastream/indexeddb", - "version": "0.3.0", + "version": "0.3.1", "description": "IndexedDB readable and writable streams for browser storage", "type": "module", "engines": { @@ -60,7 +60,7 @@ }, "homepage": "https://datastream.js.org", "dependencies": { - "@datastream/core": "0.3.0", + "@datastream/core": "0.3.1", "idb": "8.0.3" } } diff --git a/packages/ipfs/index.d.ts b/packages/ipfs/index.d.ts index 4b6e9e5..8dcd7af 100644 --- a/packages/ipfs/index.d.ts +++ b/packages/ipfs/index.d.ts @@ -1,6 +1,11 @@ // Copyright 2026 will Farrell, and datastream contributors. // SPDX-License-Identifier: MIT -import type { StreamOptions, StreamResult } from "@datastream/core"; +import type { + DatastreamReadable, + DatastreamWritable, + StreamOptions, + StreamResult, +} from "@datastream/core"; export interface IpfsNode { get(cid: string): unknown; @@ -13,7 +18,7 @@ export function ipfsGetStream( cid: string; }, streamOptions?: StreamOptions, -): Promise; +): Promise; export function ipfsAddStream( options?: { @@ -22,7 +27,7 @@ export function ipfsAddStream( }, streamOptions?: StreamOptions, ): Promise< - unknown & { + DatastreamWritable & { result: () => StreamResult; } >; diff --git a/packages/ipfs/package.json b/packages/ipfs/package.json index a74b05c..1dc7ab7 100644 --- a/packages/ipfs/package.json +++ b/packages/ipfs/package.json @@ -1,6 +1,6 @@ { "name": "@datastream/ipfs", - "version": "0.3.0", + "version": "0.3.1", "description": "IPFS get and add streaming operations", "type": "module", "engines": { @@ -61,6 +61,6 @@ }, "homepage": "https://datastream.js.org", "dependencies": { - "@datastream/core": "0.3.0" + "@datastream/core": "0.3.1" } } diff --git a/packages/json/README.md b/packages/json/README.md new file mode 100644 index 0000000..01f83cc --- /dev/null +++ b/packages/json/README.md @@ -0,0 +1,52 @@ +
+

<datastream> `json`

+ datastream logo +

JSON and NDJSON (JSON Lines) parsing and formatting transform streams.

+

+ GitHub Actions unit test status + GitHub Actions dast test status + GitHub Actions perf test status + GitHub Actions SAST test status + GitHub Actions lint test status +
+ npm version + npm install size + + npm weekly downloads + + npm provenance +
+ Open Source Security Foundation (OpenSSF) Scorecard + SLSA 3 + + Checked with Biome + Conventional Commits + + code coverage +

+

You can read the documentation at: https://datastream.js.org

+
+ + +## Install + +To install datastream you can use NPM: + +```bash +npm install --save @datastream/json +``` + + +## Documentation and examples + +For documentation and examples, refer to the main [datastream monorepo on GitHub](https://github.com/willfarrell/datastream) or [datastream official website](https://datastream.js.org). + + +## Contributing + +Everyone is very welcome to contribute to this repository. Feel free to [raise issues](https://github.com/willfarrell/datastream/issues) or to [submit Pull Requests](https://github.com/willfarrell/datastream/pulls). + + +## License + +Licensed under [MIT License](LICENSE). Copyright (c) 2026 [will Farrell](https://github.com/willfarrell), and [datastream contributors](https://github.com/willfarrell/datastream/graphs/contributors). diff --git a/packages/json/index.d.ts b/packages/json/index.d.ts index ee5fc0a..e223ced 100644 --- a/packages/json/index.d.ts +++ b/packages/json/index.d.ts @@ -1,6 +1,10 @@ // Copyright 2026 will Farrell, and datastream contributors. // SPDX-License-Identifier: MIT -import type { StreamOptions, StreamResult } from "@datastream/core"; +import type { + DatastreamTransform, + StreamOptions, + StreamResult, +} from "@datastream/core"; export interface JsonError { id: string; @@ -14,7 +18,7 @@ export function ndjsonParseStream( resultKey?: string; }, streamOptions?: StreamOptions, -): unknown & { +): DatastreamTransform> & { result: () => StreamResult>; }; @@ -24,7 +28,7 @@ export function ndjsonFormatStream( resultKey?: string; }, streamOptions?: StreamOptions, -): unknown; +): DatastreamTransform, string>; export function jsonParseStream( options?: { @@ -33,7 +37,7 @@ export function jsonParseStream( resultKey?: string; }, streamOptions?: StreamOptions, -): unknown & { +): DatastreamTransform> & { result: () => StreamResult>; }; @@ -42,4 +46,4 @@ export function jsonFormatStream( space?: number | string; }, streamOptions?: StreamOptions, -): unknown; +): DatastreamTransform, string>; diff --git a/packages/json/index.js b/packages/json/index.js index 45eae1a..66ba907 100644 --- a/packages/json/index.js +++ b/packages/json/index.js @@ -16,6 +16,11 @@ export const ndjsonParseStream = (options = {}, streamOptions = {}) => { }; const transform = (chunk, enqueue) => { + if (buffer.length + chunk.length > maxBufferSize) { + throw new Error( + `ndjsonParseStream buffer (${buffer.length + chunk.length}) exceeds maxBufferSize (${maxBufferSize})`, + ); + } buffer += chunk; let pos = 0; while (true) { @@ -34,11 +39,6 @@ export const ndjsonParseStream = (options = {}, streamOptions = {}) => { } idx++; } - if (buffer.length > maxBufferSize) { - throw new Error( - `ndjsonParseStream buffer (${buffer.length}) exceeds maxBufferSize (${maxBufferSize})`, - ); - } }; const flush = (enqueue) => { @@ -106,13 +106,13 @@ export const jsonParseStream = (options = {}, streamOptions = {}) => { }; const emitElement = (text, enqueue) => { - const trimmed = text.trim(); - if (trimmed.length === 0) return; - if (trimmed.length > maxValueSize) { + if (text.length > maxValueSize) { throw new Error( - `jsonParseStream value size (${trimmed.length}) exceeds maxValueSize (${maxValueSize})`, + `jsonParseStream value size (${text.length}) exceeds maxValueSize (${maxValueSize})`, ); } + const trimmed = text.trim(); + if (trimmed.length === 0) return; try { enqueue(JSON.parse(trimmed)); } catch { @@ -218,12 +218,12 @@ export const jsonParseStream = (options = {}, streamOptions = {}) => { }; const transform = (chunk, enqueue) => { - buffer += chunk; - if (buffer.length > maxBufferSize) { + if (buffer.length + chunk.length > maxBufferSize) { throw new Error( - `jsonParseStream buffer (${buffer.length}) exceeds maxBufferSize (${maxBufferSize})`, + `jsonParseStream buffer (${buffer.length + chunk.length}) exceeds maxBufferSize (${maxBufferSize})`, ); } + buffer += chunk; scan(enqueue); }; diff --git a/packages/json/index.test.js b/packages/json/index.test.js index 111d5df..ae89841 100644 --- a/packages/json/index.test.js +++ b/packages/json/index.test.js @@ -293,6 +293,25 @@ test(`${variant}: jsonFormatStream should handle pretty-print with space option` deepStrictEqual(combined, '[{\n "a": 1\n}\n]'); }); +// *** maxValueSize raw check *** // +test(`${variant}: jsonParseStream should enforce maxValueSize on raw value including whitespace`, async (_t) => { + const { ok, strictEqual } = await import("node:assert"); + // Raw value with padding: " 123 " = 7 chars, trimmed "123" = 3 chars + // maxValueSize: 5 — raw exceeds, trimmed does not + const padded = `[ ${"1".repeat(4)} ]`; + const streams = [ + createReadableStream(padded), + jsonParseStream({ maxValueSize: 5 }), + ]; + try { + await pipeline(streams); + throw new Error("Expected maxValueSize error"); + } catch (e) { + ok(e.message.includes("maxValueSize")); + strictEqual(e.message.includes("Expected"), false); + } +}); + test(`${variant}: jsonFormatStream roundtrip with jsonParseStream`, async (_t) => { const input = [{ a: 1 }, { b: "hello" }, { c: [1, 2, 3] }]; const streams = [ diff --git a/packages/json/index.tst.ts b/packages/json/index.tst.ts index 6c3acc2..bfb5b85 100644 --- a/packages/json/index.tst.ts +++ b/packages/json/index.tst.ts @@ -1,3 +1,5 @@ +/// +/// import type { JsonError } from "@datastream/json"; import { jsonFormatStream, diff --git a/packages/json/package.json b/packages/json/package.json index ee0b73a..1337691 100644 --- a/packages/json/package.json +++ b/packages/json/package.json @@ -1,6 +1,6 @@ { "name": "@datastream/json", - "version": "0.3.0", + "version": "0.3.1", "description": "JSON and NDJSON (JSON Lines) parsing and formatting transform streams", "type": "module", "engines": { @@ -38,7 +38,8 @@ ], "scripts": { "test": "npm run test:unit", - "test:unit": "node --test" + "test:unit": "node --test", + "test:benchmark": "node __benchmarks__/index.js" }, "license": "MIT", "keywords": [ @@ -62,6 +63,6 @@ }, "homepage": "https://datastream.js.org", "dependencies": { - "@datastream/core": "0.3.0" + "@datastream/core": "0.3.1" } } diff --git a/packages/object/index.d.ts b/packages/object/index.d.ts index ea1ff4f..1a96dc2 100644 --- a/packages/object/index.d.ts +++ b/packages/object/index.d.ts @@ -1,27 +1,34 @@ // Copyright 2026 will Farrell, and datastream contributors. // SPDX-License-Identifier: MIT -import type { StreamOptions, StreamResult } from "@datastream/core"; +import type { + DatastreamReadable, + DatastreamTransform, + DatastreamPassThrough, + StreamOptions, + StreamResult, +} from "@datastream/core"; export function objectReadableStream>( input?: T[], streamOptions?: StreamOptions, -): unknown; +): DatastreamReadable; export function objectCountStream( options?: { resultKey?: string; }, streamOptions?: StreamOptions, -): unknown & { +): DatastreamPassThrough & { result: () => StreamResult; }; export function objectBatchStream<_T = Record>( options: { keys: string[]; + maxBatchSize?: number; }, streamOptions?: StreamOptions, -): unknown; +): DatastreamTransform; export function objectPivotLongToWideStream( options: { @@ -30,7 +37,7 @@ export function objectPivotLongToWideStream( delimiter?: string; }, streamOptions?: StreamOptions, -): unknown; +): DatastreamTransform; export function objectPivotWideToLongStream( options: { @@ -40,7 +47,7 @@ export function objectPivotWideToLongStream( isNestedObject?: boolean; }, streamOptions?: StreamOptions, -): unknown; +): DatastreamTransform; export function objectKeyValueStream( options: { @@ -48,7 +55,7 @@ export function objectKeyValueStream( value: string; }, streamOptions?: StreamOptions, -): unknown; +): DatastreamTransform; export function objectKeyValuesStream( options: { @@ -56,7 +63,7 @@ export function objectKeyValuesStream( values?: string[]; }, streamOptions?: StreamOptions, -): unknown; +): DatastreamTransform; export function objectKeyJoinStream( options: { @@ -65,14 +72,14 @@ export function objectKeyJoinStream( isNestedObject?: boolean; }, streamOptions?: StreamOptions, -): unknown; +): DatastreamTransform; export function objectKeyMapStream( options: { keys: Record; }, streamOptions?: StreamOptions, -): unknown; +): DatastreamTransform; export function objectValueMapStream( options: { @@ -80,40 +87,40 @@ export function objectValueMapStream( values: Record; }, streamOptions?: StreamOptions, -): unknown; +): DatastreamTransform; export function objectPickStream( options: { keys: string[]; }, streamOptions?: StreamOptions, -): unknown; +): DatastreamTransform; export function objectOmitStream( options: { keys: string[]; }, streamOptions?: StreamOptions, -): unknown; +): DatastreamTransform; export function objectFromEntriesStream( options: { keys: string[] | (() => string[]); }, streamOptions?: StreamOptions, -): unknown; +): DatastreamTransform; export function objectToEntriesStream( options: { keys: string[] | (() => string[]); }, streamOptions?: StreamOptions, -): unknown; +): DatastreamTransform; export function objectSkipConsecutiveDuplicatesStream( - options?: Record, + options?: Record, streamOptions?: StreamOptions, -): unknown; +): DatastreamTransform; declare const _default: { readableStream: typeof objectReadableStream; diff --git a/packages/object/index.js b/packages/object/index.js index f253a35..9aa4087 100644 --- a/packages/object/index.js +++ b/packages/object/index.js @@ -20,7 +20,10 @@ export const objectCountStream = ({ resultKey } = {}, streamOptions = {}) => { return stream; }; -export const objectBatchStream = ({ keys }, streamOptions = {}) => { +export const objectBatchStream = ( + { keys, maxBatchSize = Infinity }, + streamOptions = {}, +) => { let previousId; let batch; const transform = (chunk, enqueue) => { @@ -33,6 +36,10 @@ export const objectBatchStream = ({ keys }, streamOptions = {}) => { batch = []; } batch.push(chunk); + if (batch.length >= maxBatchSize) { + enqueue(batch); + batch = []; + } }; const flush = (enqueue) => { if (batch) { @@ -52,7 +59,7 @@ export const objectPivotLongToWideStream = ( if (!Array.isArray(chunks)) { throw new Error("Expected chunk to be array, use with objectBatchStream"); } - const row = chunks[0]; + const row = { ...chunks[0] }; for (const chunk of chunks) { const keyParam = keys.map((key) => chunk[key]).join(delimiter); diff --git a/packages/object/index.test.js b/packages/object/index.test.js index 5eaf66b..6c0a741 100644 --- a/packages/object/index.test.js +++ b/packages/object/index.test.js @@ -534,3 +534,48 @@ test(`${variant}: objectKeyJoinStream should not mutate input chunks`, async (_t await streamToArray(stream); deepStrictEqual(input[0], original); }); + +// *** objectBatchStream maxBatchSize *** // +test(`${variant}: objectBatchStream should enforce maxBatchSize`, async (_t) => { + const { ok } = await import("node:assert"); + const input = Array.from({ length: 10 }, (_, i) => ({ a: "same", b: i })); + const streams = [ + createReadableStream(input), + objectBatchStream({ keys: ["a"], maxBatchSize: 3 }), + ]; + + const stream = pipejoin(streams); + const output = await streamToArray(stream); + + // All items share key "same", but batches should be split at size 3 + for (const batch of output) { + ok(batch.length <= 3); + } + strictEqual( + output.reduce((sum, b) => sum + b.length, 0), + 10, + ); +}); + +// *** objectPivotLongToWideStream should not mutate input *** // +test(`${variant}: objectPivotLongToWideStream should not mutate input chunks`, async (_t) => { + const input = [ + [ + { region: "US", metric: "sales", value: 100 }, + { region: "US", metric: "cost", value: 50 }, + ], + ]; + const originalFirst = { ...input[0][0] }; + const streams = [ + createReadableStream(input), + objectPivotLongToWideStream({ + keys: ["metric"], + valueParam: "value", + }), + ]; + const stream = pipejoin(streams); + await streamToArray(stream); + + // Original input[0][0] should be unchanged + deepStrictEqual(input[0][0], originalFirst); +}); diff --git a/packages/object/index.tst.ts b/packages/object/index.tst.ts index 3473084..7d11a28 100644 --- a/packages/object/index.tst.ts +++ b/packages/object/index.tst.ts @@ -1,4 +1,5 @@ -import type { StreamResult } from "@datastream/core"; +/// +/// import { objectBatchStream, objectCountStream, @@ -31,7 +32,7 @@ describe("objectReadableStream", () => { describe("objectCountStream", () => { test("returns stream with result", () => { const stream = objectCountStream(); - expect(stream.result()).type.toBe>(); + expect(stream.result()).type.not.toBeAssignableTo(); }); test("accepts resultKey", () => { diff --git a/packages/object/package.json b/packages/object/package.json index 81aea1b..ebe35e2 100644 --- a/packages/object/package.json +++ b/packages/object/package.json @@ -1,6 +1,6 @@ { "name": "@datastream/object", - "version": "0.3.0", + "version": "0.3.1", "description": "Object transform streams for picking, omitting, pivoting, batching, and key mapping", "type": "module", "engines": { @@ -60,6 +60,6 @@ }, "homepage": "https://datastream.js.org", "dependencies": { - "@datastream/core": "0.3.0" + "@datastream/core": "0.3.1" } } diff --git a/packages/string/index.d.ts b/packages/string/index.d.ts index cb7a95a..e948a00 100644 --- a/packages/string/index.d.ts +++ b/packages/string/index.d.ts @@ -1,18 +1,24 @@ // Copyright 2026 will Farrell, and datastream contributors. // SPDX-License-Identifier: MIT -import type { StreamOptions, StreamResult } from "@datastream/core"; +import type { + DatastreamReadable, + DatastreamTransform, + DatastreamPassThrough, + StreamOptions, + StreamResult, +} from "@datastream/core"; export function stringReadableStream( input: string | string[], streamOptions?: StreamOptions, -): unknown; +): DatastreamReadable; export function stringLengthStream( options?: { resultKey?: string; }, streamOptions?: StreamOptions, -): unknown & { +): DatastreamPassThrough & { result: () => StreamResult; }; @@ -22,7 +28,7 @@ export function stringCountStream( resultKey?: string; }, streamOptions?: StreamOptions, -): unknown & { +): DatastreamPassThrough & { result: () => StreamResult; }; @@ -31,19 +37,19 @@ export function stringMinimumFirstChunkSize( chunkSize?: number; }, streamOptions?: StreamOptions, -): unknown; +): DatastreamTransform; export function stringMinimumChunkSize( options?: { chunkSize?: number; }, streamOptions?: StreamOptions, -): unknown; +): DatastreamTransform; export function stringSkipConsecutiveDuplicates( - options?: Record, + options?: Record, streamOptions?: StreamOptions, -): unknown; +): DatastreamTransform; export function stringReplaceStream( options: { @@ -52,7 +58,7 @@ export function stringReplaceStream( maxBufferSize?: number; }, streamOptions?: StreamOptions, -): unknown; +): DatastreamTransform; export function stringSplitStream( options: { @@ -60,7 +66,7 @@ export function stringSplitStream( maxBufferSize?: number; }, streamOptions?: StreamOptions, -): unknown; +): DatastreamTransform; declare const _default: { readableStream: typeof stringReadableStream; diff --git a/packages/string/index.js b/packages/string/index.js index f53d9cc..410d0d6 100644 --- a/packages/string/index.js +++ b/packages/string/index.js @@ -106,9 +106,22 @@ export const stringReplaceStream = (options, streamOptions = {}) => { replacement, maxBufferSize = 16_777_216, // 16MB } = options; + if ( + pattern instanceof RegExp && + !pattern.flags.includes("g") && + !pattern.flags.includes("y") + ) { + throw new Error( + "RegExp pattern must include the global (g) or sticky (y) flag", + ); + } let previousChunk = ""; + const useReplaceAll = typeof pattern === "string"; const transform = (chunk, enqueue) => { - const newChunk = (previousChunk + chunk).replace(pattern, replacement); + const combined = previousChunk + chunk; + const newChunk = useReplaceAll + ? combined.replaceAll(pattern, replacement) + : combined.replace(pattern, replacement); enqueue(newChunk.substring(0, previousChunk.length)); previousChunk = newChunk.substring(previousChunk.length); if (previousChunk.length > maxBufferSize) { diff --git a/packages/string/index.tst.ts b/packages/string/index.tst.ts index ca7b197..6a34dd0 100644 --- a/packages/string/index.tst.ts +++ b/packages/string/index.tst.ts @@ -1,4 +1,5 @@ -import type { StreamResult } from "@datastream/core"; +/// +/// import { stringCountStream, stringLengthStream, @@ -24,7 +25,7 @@ describe("stringLengthStream", () => { test("returns stream with result method", () => { const stream = stringLengthStream(); expect(stream.result).type.not.toBeAssignableTo(); - expect(stream.result()).type.toBe>(); + expect(stream.result()).type.not.toBeAssignableTo(); }); test("accepts resultKey option", () => { @@ -37,7 +38,7 @@ describe("stringLengthStream", () => { describe("stringCountStream", () => { test("returns stream with result method", () => { const stream = stringCountStream({ substr: "x" }); - expect(stream.result()).type.toBe>(); + expect(stream.result()).type.not.toBeAssignableTo(); }); }); diff --git a/packages/string/package.json b/packages/string/package.json index 0507619..c50e388 100644 --- a/packages/string/package.json +++ b/packages/string/package.json @@ -1,6 +1,6 @@ { "name": "@datastream/string", - "version": "0.3.0", + "version": "0.3.1", "description": "String transform streams for splitting, replacing, counting, and deduplication", "type": "module", "engines": { @@ -60,6 +60,6 @@ }, "homepage": "https://datastream.js.org", "dependencies": { - "@datastream/core": "0.3.0" + "@datastream/core": "0.3.1" } } diff --git a/packages/validate/index.d.ts b/packages/validate/index.d.ts index 4da3d0f..dc22d2e 100644 --- a/packages/validate/index.d.ts +++ b/packages/validate/index.d.ts @@ -1,6 +1,10 @@ // Copyright 2026 will Farrell, and datastream contributors. // SPDX-License-Identifier: MIT -import type { StreamOptions, StreamResult } from "@datastream/core"; +import type { + DatastreamTransform, + StreamOptions, + StreamResult, +} from "@datastream/core"; export interface ValidateError { id: string; @@ -21,9 +25,10 @@ export function validateStream( onErrorEnqueue?: boolean; allowCoerceTypes?: boolean; resultKey?: string; + maxErrorRows?: number; }, streamOptions?: StreamOptions, -): unknown & { +): DatastreamTransform & { result: () => StreamResult>; }; diff --git a/packages/validate/index.js b/packages/validate/index.js index 7c4867b..56b65fb 100644 --- a/packages/validate/index.js +++ b/packages/validate/index.js @@ -19,7 +19,14 @@ export const transpileSchema = (schema, ajvOptions) => { }; export const validateStream = ( - { schema, idxStart, onErrorEnqueue, allowCoerceTypes, resultKey }, + { + schema, + idxStart, + onErrorEnqueue, + allowCoerceTypes, + resultKey, + maxErrorRows = Infinity, + }, streamOptions = {}, ) => { idxStart ??= 0; @@ -42,7 +49,9 @@ export const validateStream = ( if (!value[id]) { value[id] = { id, keys, message, idx: [] }; } - value[id].idx.push(idx); + if (value[id].idx.length < maxErrorRows) { + value[id].idx.push(idx); + } } } if (chunkValid || onErrorEnqueue) { diff --git a/packages/validate/index.test.js b/packages/validate/index.test.js index 873f08b..5f3eefc 100644 --- a/packages/validate/index.test.js +++ b/packages/validate/index.test.js @@ -426,6 +426,26 @@ test(`${variant}: validateStream should handle pre-compiled schema with no messa strictEqual(result.validate[errorKey].message, ""); }); +// *** maxErrorRows *** // +test(`${variant}: validateStream should cap error indices with maxErrorRows`, async (_t) => { + const input = Array.from({ length: 100 }, (_, i) => ({ a: `bad${i}` })); + const schema = { + type: "object", + properties: { + a: { type: "number" }, + }, + }; + + const streams = [ + createReadableStream(input), + validateStream({ schema, maxErrorRows: 10 }), + ]; + const result = await pipeline(streams); + + const errorKey = Object.keys(result.validate)[0]; + ok(result.validate[errorKey].idx.length <= 10); +}); + // *** default export *** // test(`${variant}: default export should be validateStream`, (_t) => { strictEqual(validateDefault, validateStream); diff --git a/packages/validate/index.tst.ts b/packages/validate/index.tst.ts index ff00992..88f5dc8 100644 --- a/packages/validate/index.tst.ts +++ b/packages/validate/index.tst.ts @@ -1,3 +1,5 @@ +/// +/// import type { ValidateError } from "@datastream/validate"; import { transpileSchema, validateStream } from "@datastream/validate"; import { describe, expect, test } from "tstyche"; diff --git a/packages/validate/package.json b/packages/validate/package.json index ab1a5a3..087ef37 100644 --- a/packages/validate/package.json +++ b/packages/validate/package.json @@ -1,6 +1,6 @@ { "name": "@datastream/validate", - "version": "0.3.0", + "version": "0.3.1", "description": "JSON Schema validation transform streams using Ajv", "type": "module", "engines": { @@ -60,7 +60,7 @@ }, "homepage": "https://datastream.js.org", "dependencies": { - "@datastream/core": "0.3.0", + "@datastream/core": "0.3.1", "ajv-cmd": "0.11.0" } } diff --git a/websites/datastream.js.org/package.json b/websites/datastream.js.org/package.json index 7d864dd..01cf2f8 100644 --- a/websites/datastream.js.org/package.json +++ b/websites/datastream.js.org/package.json @@ -2,7 +2,7 @@ "name": "datastream.js.org", "description": "SvelteKit SSR", "private": true, - "version": "0.3.0", + "version": "0.3.1", "type": "module", "scripts": { "start": "vite dev", diff --git a/websites/datastream.js.org/src/routes/docs/packages/base64/+page.md b/websites/datastream.js.org/src/routes/docs/packages/base64/+page.md index cde23c4..ac9f128 100644 --- a/websites/datastream.js.org/src/routes/docs/packages/base64/+page.md +++ b/websites/datastream.js.org/src/routes/docs/packages/base64/+page.md @@ -15,6 +15,8 @@ npm install @datastream/base64 Encodes data to base64. Handles chunk boundaries correctly by buffering partial 3-byte groups. +> **Note:** Input must be Latin1-encoded strings (code points 0-255). Use `charsetEncodeStream` to convert other encodings before base64 encoding. + ### Example ```javascript diff --git a/websites/datastream.js.org/src/routes/docs/packages/compress/+page.md b/websites/datastream.js.org/src/routes/docs/packages/compress/+page.md index a0b3d1c..76e43f7 100644 --- a/websites/datastream.js.org/src/routes/docs/packages/compress/+page.md +++ b/websites/datastream.js.org/src/routes/docs/packages/compress/+page.md @@ -18,6 +18,7 @@ npm install @datastream/compress | Option | Type | Default | Description | |--------|------|---------|-------------| | `quality` | `number` | `-1` | Compression level (-1 to 9). -1 = default, 0 = none, 9 = best | +| `maxOutputSize` | `number` | — | Maximum compressed output in bytes (web variant) | ### `gzipDecompressStream` Transform @@ -54,6 +55,7 @@ await pipeline([ | Option | Type | Default | Description | |--------|------|---------|-------------| | `quality` | `number` | `-1` | Compression level (-1 to 9) | +| `maxOutputSize` | `number` | — | Maximum compressed output in bytes (web variant) | ### `deflateDecompressStream` Transform @@ -91,7 +93,9 @@ Requires Node.js with zstd support. |--------|------|---------|-------------| | `maxOutputSize` | `number` | — | Maximum decompressed output in bytes. Destroys the stream with an error when exceeded | -## Decompression bomb protection +## Output size protection + +### Decompression bombs A malicious compressed payload known as a "decompression bomb" can be as small as a few kilobytes but expand to gigabytes when decompressed, exhausting memory and crashing the process. Setting `maxOutputSize` ensures decompression is aborted before memory is exhausted. Always set this when decompressing untrusted input. @@ -102,6 +106,17 @@ import { gzipDecompressStream } from '@datastream/compress' gzipDecompressStream({ maxOutputSize: 100 * 1024 * 1024 }) ``` +### Compression output limits + +Compression streams also support `maxOutputSize` (web variant) to cap compressed output size. This can be useful to enforce storage limits. + +```javascript +import { gzipCompressStream } from '@datastream/compress' + +// Limit compressed output to 50MB +gzipCompressStream({ maxOutputSize: 50 * 1024 * 1024 }) +``` + ## Platform support | Algorithm | Node.js | Browser | diff --git a/websites/datastream.js.org/src/routes/docs/packages/core/+page.md b/websites/datastream.js.org/src/routes/docs/packages/core/+page.md index 864de49..ba87438 100644 --- a/websites/datastream.js.org/src/routes/docs/packages/core/+page.md +++ b/websites/datastream.js.org/src/routes/docs/packages/core/+page.md @@ -135,6 +135,26 @@ async function* generate() { const stream = createReadableStream(generate()) ``` +### `createReadableStreamFromString(input, streamOptions)` Readable + +Creates a Readable stream from a string, chunking it at `chunkSize` (default 16KB). Useful when you need explicit control over string chunking separate from `createReadableStream`. + +```javascript +import { createReadableStreamFromString } from '@datastream/core' + +const stream = createReadableStreamFromString(largeString, { chunkSize: 4096 }) +``` + +### `createReadableStreamFromArrayBuffer(input, streamOptions)` Readable + +Creates a Readable stream from an `ArrayBuffer` or `Uint8Array`, chunking it at `chunkSize` (default 16KB). + +```javascript +import { createReadableStreamFromArrayBuffer } from '@datastream/core' + +const stream = createReadableStreamFromArrayBuffer(buffer, { chunkSize: 8192 }) +``` + ### `createPassThroughStream(fn, flush?, streamOptions)` Transform (PassThrough) Creates a stream that observes each chunk without modifying it. The chunk is automatically passed through.