From 59e454a655e44e1e98697fbf05371914c6bc6147 Mon Sep 17 00:00:00 2001 From: Derrick Williams Date: Fri, 30 May 2025 17:32:57 +0000 Subject: [PATCH 1/2] add extra_transforms block documentation to chain transform documentation --- .../content/en/documentation/sdks/yaml.md | 45 +++++++++++++++++++ 1 file changed, 45 insertions(+) diff --git a/website/www/site/content/en/documentation/sdks/yaml.md b/website/www/site/content/en/documentation/sdks/yaml.md index 3d266ade6160..b8b7c1ba346d 100644 --- a/website/www/site/content/en/documentation/sdks/yaml.md +++ b/website/www/site/content/en/documentation/sdks/yaml.md @@ -273,6 +273,51 @@ pipeline: path: /path/to/output.json ``` +If a `chain` pipeline has required error consumption or needs additional +transforms not supported in a `chain` context, use an `extra_transforms` block. + +``` +pipeline: + type: chain + transforms: + - type: ReadFromCsv + config: + path: /path/to/input*.csv + + - type: MapToFields + name: SomeStep + config: + language: python + fields: + col1: col1 + # This could raise a divide-by-zero error. + ratio: col2 / col3 + error_handling: + output: errors + + - type: MapToFields + name: AnotherStep + config: + language: python + fields: + col1: col1 + # This could raise a divide-by-zero error. + inverse_ratio: 1 / ratio + error_handling: + output: errors + + - type: WriteToJson + config: + path: /path/to/output.json + + extra_transforms: + - type: WriteToJson + name: WriteErrors + input: [SomeStep.errors, AnotherStep.errors] + config: + path: /path/to/errors.json +``` + ### Source and sink transforms As syntactic sugar, you can name the first and last transforms in your pipeline From 9147cde25b945cc43e3a6fffa550762b4c3d2edb Mon Sep 17 00:00:00 2001 From: Derrick Williams Date: Fri, 30 May 2025 17:36:41 +0000 Subject: [PATCH 2/2] add more comment --- website/www/site/content/en/documentation/sdks/yaml.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/website/www/site/content/en/documentation/sdks/yaml.md b/website/www/site/content/en/documentation/sdks/yaml.md index b8b7c1ba346d..73d1eebaae95 100644 --- a/website/www/site/content/en/documentation/sdks/yaml.md +++ b/website/www/site/content/en/documentation/sdks/yaml.md @@ -274,7 +274,8 @@ pipeline: ``` If a `chain` pipeline has required error consumption or needs additional -transforms not supported in a `chain` context, use an `extra_transforms` block. +transforms not supported in a typical `chain` context, use an +`extra_transforms` block. ``` pipeline: