diff --git a/docs/docs/Media-Selectors-Guide.md b/docs/docs/Media-Selectors-Guide.md
index fd130a4c061a..1122580f8c74 100644
--- a/docs/docs/Media-Selectors-Guide.md
+++ b/docs/docs/Media-Selectors-Guide.md
@@ -11,8 +11,9 @@ produced.
# New Job Request Fields
-Below is an example of a job that uses media selectors. The job uses a two-stage pipeline.
-The first stage performs language identification. The second performs translation.
+Below is an example of a job that uses `JSON_PATH` media selectors. The job uses a two-stage
+pipeline. The first stage performs language identification. The second performs translation.
+
```json
{
"algorithmProperties": {},
@@ -43,6 +44,7 @@ The first stage performs language identification. The second performs translatio
"priority": 4
}
```
+
- `$.media.*.mediaSelectorsOutputAction`: Name of the action that produces content for the media
selectors output file. In the above example, we specify that we want the translated content
from Argos in the media selectors output file rather than the detected language from the first
@@ -68,18 +70,20 @@ The first stage performs language identification. The second performs translatio
of the document will be replaced with the action output. When provided, the selected parts of
the document will contain the original content, followed by the value of this property, and
finally the action output.
+
- `MEDIA_SELECTORS_DUPLICATE_POLICY`: Specifies how to handle the case where a job uses media
selectors and there are multiple outputs for a single selection. When set to `LONGEST`, the
longer of the two outputs is chosen and the shorter one is discarded. When set to `ERROR`,
duplicates are considered an error. When set to `JOIN`, the duplicates are combined using
` | ` as a delimiter.
+
- `MEDIA_SELECTORS_NO_MATCHES_IS_ERROR`: When true and a job uses media selectors, an error will be
generated when none of the selectors match content from the media.
# Media Selector Types
-`JSON_PATH` is only type currently supported, but others are planned.
+[`JSON_PATH`](#json_path) and [`CSV_COLS`](#csv_cols) are currently supported.
## JSON_PATH
@@ -94,6 +98,7 @@ other JSON types are ignored. When the JsonPath expression matches an array, eac
recursively explored. When the expression matches an object, keys are left unchanged and each value
of the object is recursively explored.
+
### JSON_PATH Matching Example
```json
@@ -122,14 +127,14 @@ Expression | Matches
`$.key2.key3.*.*[0]` | d, f
-
-# Media Selectors Output File
+### JSON_PATH Output File
When media selectors are used, the JsonOutputObject will contain a URI referencing the file
location in the `$.media.*.mediaSelectorsOutputUri` field.
-The job from the [New Job Request Fields section](#new-job-request-fields) could be used with the
-document below.
+For example, consider that the `mediaUri` from the job in the
+[New Job Request Fields section](#new-job-request-fields) refers to the document below.
+
```json
{
"otherStuffKey": ["other stuff value"],
@@ -166,6 +171,7 @@ document below.
```
The `mediaSelectorsOutputUri` field will refer to a document containing the content below.
+
```json
{
"otherStuffKey": ["other stuff value"],
@@ -238,3 +244,213 @@ below.
]
}
```
+
+
+## CSV_COLS
+
+Used to extract content from specific columns of a CSV file. The expression itself must be a
+single row of CSV listing the columns to extract. The `CSV_SELECTORS_ARE_INDICES` job property
+controls whether the entries refer to column names or zero-based integer indices.
+
+
+### CSV-Specific Job Properties
+
+- `CSV_SELECTORS_ARE_INDICES`: When `FALSE` (the default), the selector expression must contain
+ column names. When `TRUE` the selector should contain the zero-based integer indices of the
+ columns that should be processed.
+
+- `CSV_CSV_FIRST_ROW_IS_DATA`: When `FALSE` (the default), the first row is considered headers and
+ will not be processed. When `TRUE`, the first row is considered data and the first row will be
+ processed.
+
+An issue when processing CSV is that sometimes the first row is considered headers (a.k.a. column
+names) and in others the first row is actually data and there are no headers.
+
+- In the default configuration (`CSV_SELECTORS_ARE_INDICES` = `FALSE` and
+ `CSV_FIRST_ROW_IS_DATA` = `FALSE`), the selector expression refers to column names and the
+ first row is not processed as data.
+
+- If the first row is actual data, and you want to specify the columns by index instead of by
+ name in the selector expression, set `CSV_SELECTORS_ARE_INDICES` = `TRUE` and
+ `CSV_FIRST_ROW_IS_DATA` = `TRUE`.
+
+- If the first row is headers, and you want to specify the columns by index instead of by
+ name in the selector expression, set `CSV_SELECTORS_ARE_INDICES` = `TRUE` and
+ `CSV_FIRST_ROW_IS_DATA` = `FALSE`.
+
+
+### CSV_COLS Matching Example
+
+The table below shows combinations of values for `CSV_SELECTORS_ARE_INDICES` and
+`CSV_FIRST_ROW_IS_DATA` when matched against this CSV content:
+
+```text
+header0,header1,"header,2"
+a,b,c
+d,e,f,g
+```
+
+Expression | `CSV_SELECTORS_`
`ARE_INDICES` | `CSV_FIRST_ROW_`
`IS_DATA` | Matches
+---------------------|---------------|-------------------|------------
+`header0,"header,2"` | FALSE | FALSE | a, c, d, f
+`header0,"header,2"` | FALSE | TRUE | header0, "header,2", a, c, d, f
+`header0,headerX` | FALSE | TRUE / FALSE | **Error**: "headerX" does not exist
+`header0,header,2` | FALSE | TRUE / FALSE | **Error**: "header" and "2" do not exist
+`header0,"header,2"` | TRUE | TRUE / FALSE | **Error**: The expression contains non-integers.
+`0,2` | TRUE | FALSE | a, c, d, f
+`0,2` | TRUE | TRUE | header0, "header,2", a, c, d, f
+`0,3,4` | TRUE | FALSE | a, d, g
+`0,2` | FALSE | TRUE / FALSE | **Error**: There are no columns with "0" or "2" as the header.
+
+
+### CSV Text Encodings
+
+We recommend submitting UTF-8-encoded CSV files, but we do attempt to recognize other text
+encodings. When attempting to determine the input file encoding, the Workflow Manager will inspect
+the first 12,000 bytes of the file. If all of the 12,000 bytes are valid UTF-8 bytes, then the
+Workflow Manager will treat the file as UTF-8. Otherwise, the Workflow Manager will use
+[Tika's `CharsetDetector`](https://tika.apache.org/2.9.1/api/org/apache/tika/parser/txt/CharsetDetector.html)
+to determine the encoding.
+
+The media selectors output file will always be UTF-8-encoded. If the input file was UTF-8-encoded
+and had a byte-order mark, then a byte-order mark will be added to the output file.
+
+
+#### Byte-order mark
+
+The UTF-8, UTF-16, and UTF-32 text encodings may have a byte-order mark present. The byte-order
+mark is the Unicode character named "ZERO WIDTH NO-BREAK SPACE" with a code point of U+FEFF. Each
+encoding will encode it as different bytes. For example, in UTF-8 it is encoded with three bytes:
+`0xEF`, `0xBB`, `0xBF`.
+
+Many CSV parsers do not have special handling for the byte-order mark. They just treat it as a
+normal character and consider it to be the first character in the first cell. The Workflow Manager
+discards the byte-order mark before parsing the CSV.
+
+
+#### Excel
+
+- If you open a CSV file in Microsoft Excel and the text is garbled, you should open the file
+ in a text editor that supports UTF-8 and see if the text is garbled there too.
+
+- When saving a CSV file from Excel, if you select "CSV (Comma delimited)(*.csv)", Excel will
+ silently replace East Asian characters with question marks. Selecting
+ "CSV UTF-8 (Comma delimited) (.csv)" preserves the East Asian characters, but it adds a
+ byte-order mark to the file.
+
+- If you open a UTF-8-encoded file in Excel, it will treat it as ISO-8859-1 unless the file has
+ a UTF-8 byte-order mark.
+
+Using a byte-order mark with UTF-8 is uncommon because the UTF-8 encoding does not have endianess
+like UTF-16 and UTF-32. The byte-order mark added by Excel can cause problems because a
+lot of software does not expect it to be present.
+
+
+As an example, consider an Excel spreadsheet with the following content:
+
+Col,1 | Col,2
+------|--------
+item1 | item2
+
+
+If you save that as "CSV UTF-8 (Comma delimited) (.csv)" and then `cat` the file you will get:
+
+```text
+"Col,1","Col,2"
+item1,item2
+```
+
+Since the two column names contain commas, the cells need to be escaped with quotation marks so that
+the internal comma is not interpreted as a record separator. If you parse that file with
+[Python's built-in CSV parser](https://docs.python.org/3/library/csv.html) you get the following
+result:
+
+ "Col | 1" | Col,2
+-------|-------|------
+item1 | item2 |
+
+
+While the first cell above appears to be four characters in length, it is actually five. The first
+character is U+FEFF (ZERO WIDTH NO-BREAK SPACE). Since the first cell starts with
+U+FEFF (ZERO WIDTH NO-BREAK SPACE), rather than U+0022 (QUOTATION MARK), the first comma is
+unescaped so it is interpreted as the record separator.
+
+Since the byte-order mark is invisible when rendered, simply printing the content will not reveal
+the issue. It is visible in a hexdump produced with `hexdump -C` on Linux or `Format-Hex` in
+Windows Powershell. In the output from `hexdump -C` below, you can see that there is content before
+the quotation mark and that content matches the UTF-8-encoded byte-order mark of `0xEF`, `0xBB`,
+`0xBF`.
+
+```text
+00000000 ef bb bf 22 43 6f 6c 2c 31 22 2c 22 43 6f 6c 2c |..."Col,1","Col,|
+00000010 32 22 0d 0a 69 74 65 6d 31 2c 69 74 65 6d 32 0d |2"..item1,item2.|
+00000020 0a |.|
+```
+
+
+### CSV_COLS Output File
+
+When media selectors are used, a copy of the input file with the specified sections replaced by
+component output is produced. The URI to the file will be present in the
+`$.media.*.mediaSelectorsOutputUri` field.
+
+Below is an example of a job that uses `CSV_COLS` media selectors. The job uses a two-stage
+pipeline. The first stage performs language identification. The second performs translation.
+
+```json
+{
+ "algorithmProperties": {},
+ "buildOutput": true,
+ "jobProperties": {},
+ "media": [
+ {
+ "mediaUri": "file:///opt/mpf/share/remote-media/test-csv-translation.csv",
+ "properties": {},
+ "mediaSelectorsOutputAction": "ARGOS TRANSLATION (WITH FF REGION AND NO TASK MERGING) ACTION",
+ "mediaSelectors": [
+ {
+ "type": "CSV_COLS",
+ "expression": "Spanish",
+ "resultDetectionProperty": "TRANSLATION",
+ "selectionProperties": {}
+ },
+ {
+ "type": "CSV_COLS",
+ "expression": "Chinese",
+ "resultDetectionProperty": "TRANSLATION",
+ "selectionProperties": {}
+ }
+ ]
+ }
+ ],
+ "pipelineName": "ARGOS TRANSLATION (WITH FASTTEXT LANGUAGE ID) TEXT FILE PIPELINE",
+ "priority": 4
+}
+```
+
+The input file, `test-csv-translation.csv`, contains the content below.
+```text
+English,Spanish,Chinese
+"Hello, how are you?","¿Hola, cómo estás?",你好吗？
+Where is the library?,¿Dónde está la biblioteca?,图书馆在哪里？
+What time is it?,¿Qué hora es?,现在是几奌？
+```
+
+The `mediaSelectorsOutputUri` field from the output object will refer to a document containing the
+content below.
+```text
+English,Spanish,Chinese
+"Hello, how are you?","Hello, how are you?",How are you?
+Where is the library?,Where's the library?,Where's the library?
+What time is it?,What time is it?,What time is it?
+```
+
+If `MEDIA_SELECTORS_DELIMETER` was set to " | Translation: ", the file would contain the content
+below.
+
+```text
+English,Spanish,Chinese
+"Hello, how are you?","¿Hola, cómo estás? | Translation: Hello, how are you?",你好吗？ | Translation: How are you?
+Where is the library?,¿Dónde está la biblioteca? | Translation: Where's the library?,图书馆在哪里？ | Translation: Where's the library?
+What time is it?,¿Qué hora es? | Translation: What time is it?,现在是几奌？ | Translation: What time is it?
+```
diff --git a/docs/site/Media-Selectors-Guide/index.html b/docs/site/Media-Selectors-Guide/index.html
index fad6240444d5..9ce87afcec48 100644
--- a/docs/site/Media-Selectors-Guide/index.html
+++ b/docs/site/Media-Selectors-Guide/index.html
@@ -155,12 +155,11 @@
JSON_PATH
+ CSV_COLS
+
- Media Selectors Output File
-
-
@@ -272,8 +271,8 @@
processed. A copy of the input file with the specified sections replaced by component output is
produced.
New Job Request Fields
-Below is an example of a job that uses media selectors. The job uses a two-stage pipeline.
-The first stage performs language identification. The second performs translation.
+Below is an example of a job that uses JSON_PATH media selectors. The job uses a two-stage
+pipeline. The first stage performs language identification. The second performs translation.
{
"algorithmProperties": {},
"buildOutput": true,
@@ -324,20 +323,26 @@ New Job Request Fields
New Job Properties
-MEDIA_SELECTORS_DELIMETER: When not provided and a job uses media selectors, the selected parts
+-
+
MEDIA_SELECTORS_DELIMETER: When not provided and a job uses media selectors, the selected parts
of the document will be replaced with the action output. When provided, the selected parts of
the document will contain the original content, followed by the value of this property, and
- finally the action output.
-MEDIA_SELECTORS_DUPLICATE_POLICY: Specifies how to handle the case where a job uses media
+ finally the action output.
+-
+
MEDIA_SELECTORS_DUPLICATE_POLICY: Specifies how to handle the case where a job uses media
selectors and there are multiple outputs for a single selection. When set to LONGEST, the
longer of the two outputs is chosen and the shorter one is discarded. When set to ERROR,
duplicates are considered an error. When set to JOIN, the duplicates are combined using
- | as a delimiter.
-MEDIA_SELECTORS_NO_MATCHES_IS_ERROR: When true and a job uses media selectors, an error will be
- generated when none of the selectors match content from the media.| as a delimiter.
+
+
+MEDIA_SELECTORS_NO_MATCHES_IS_ERROR: When true and a job uses media selectors, an error will be
+ generated when none of the selectors match content from the media.
+
-JSON_PATH is only type currently supported, but others are planned.
+JSON_PATH and CSV_COLS are currently supported.
JSON_PATH
Used to extract content for JSON files. Uses the "Jayway JsonPath" library to parse the expressions.
The specific syntax supported is available on their
@@ -403,11 +408,11 @@
JSON_PATH Matching Example
-
+JSON_PATH Output File
When media selectors are used, the JsonOutputObject will contain a URI referencing the file
location in the $.media.*.mediaSelectorsOutputUri field.
-The job from the New Job Request Fields section could be used with the
-document below.
+For example, consider that the mediaUri from the job in the
+New Job Request Fields section refers to the document below.
{
"otherStuffKey": ["other stuff value"],
"spanishMessages": [
@@ -509,6 +514,259 @@
}
]
}
+
+CSV_COLS
+Used to extract content from specific columns of a CSV file. The expression itself must be a
+single row of CSV listing the columns to extract. The CSV_SELECTORS_ARE_INDICES job property
+controls whether the entries refer to column names or zero-based integer indices.
+CSV-Specific Job Properties
+
+-
+
CSV_SELECTORS_ARE_INDICES: When FALSE (the default), the selector expression must contain
+ column names. When TRUE the selector should contain the zero-based integer indices of the
+ columns that should be processed.
+
+-
+
CSV_CSV_FIRST_ROW_IS_DATA: When FALSE (the default), the first row is considered headers and
+ will not be processed. When TRUE, the first row is considered data and the first row will be
+ processed.
+
+
+An issue when processing CSV is that sometimes the first row is considered headers (a.k.a. column
+names) and in others the first row is actually data and there are no headers.
+
+-
+
In the default configuration (CSV_SELECTORS_ARE_INDICES = FALSE and
+ CSV_FIRST_ROW_IS_DATA = FALSE), the selector expression refers to column names and the
+ first row is not processed as data.
+
+-
+
If the first row is actual data, and you want to specify the columns by index instead of by
+ name in the selector expression, set CSV_SELECTORS_ARE_INDICES = TRUE and
+ CSV_FIRST_ROW_IS_DATA = TRUE.
+
+-
+
If the first row is headers, and you want to specify the columns by index instead of by
+ name in the selector expression, set CSV_SELECTORS_ARE_INDICES = TRUE and
+ CSV_FIRST_ROW_IS_DATA = FALSE.
+
+
+CSV_COLS Matching Example
+The table below shows combinations of values for CSV_SELECTORS_ARE_INDICES and
+CSV_FIRST_ROW_IS_DATA when matched against this CSV content:
+header0,header1,"header,2"
+a,b,c
+d,e,f,g
+
+
+
+
+| Expression |
+CSV_SELECTORS_
ARE_INDICES |
+CSV_FIRST_ROW_
IS_DATA |
+Matches |
+
+
+
+
+header0,"header,2" |
+FALSE |
+FALSE |
+a, c, d, f |
+
+
+header0,"header,2" |
+FALSE |
+TRUE |
+header0, "header,2", a, c, d, f |
+
+
+header0,headerX |
+FALSE |
+TRUE / FALSE |
+Error: "headerX" does not exist |
+
+
+header0,header,2 |
+FALSE |
+TRUE / FALSE |
+Error: "header" and "2" do not exist |
+
+
+header0,"header,2" |
+TRUE |
+TRUE / FALSE |
+Error: The expression contains non-integers. |
+
+
+0,2 |
+TRUE |
+FALSE |
+a, c, d, f |
+
+
+0,2 |
+TRUE |
+TRUE |
+header0, "header,2", a, c, d, f |
+
+
+0,3,4 |
+TRUE |
+FALSE |
+a, d, g |
+
+
+0,2 |
+FALSE |
+TRUE / FALSE |
+Error: There are no columns with "0" or "2" as the header. |
+
+
+
+CSV Text Encodings
+We recommend submitting UTF-8-encoded CSV files, but we do attempt to recognize other text
+encodings. When attempting to determine the input file encoding, the Workflow Manager will inspect
+the first 12,000 bytes of the file. If all of the 12,000 bytes are valid UTF-8 bytes, then the
+Workflow Manager will treat the file as UTF-8. Otherwise, the Workflow Manager will use
+Tika's CharsetDetector
+to determine the encoding.
+The media selectors output file will always be UTF-8-encoded. If the input file was UTF-8-encoded
+and had a byte-order mark, then a byte-order mark will be added to the output file.
+Byte-order mark
+The UTF-8, UTF-16, and UTF-32 text encodings may have a byte-order mark present. The byte-order
+mark is the Unicode character named "ZERO WIDTH NO-BREAK SPACE" with a code point of U+FEFF. Each
+encoding will encode it as different bytes. For example, in UTF-8 it is encoded with three bytes:
+0xEF, 0xBB, 0xBF.
+Many CSV parsers do not have special handling for the byte-order mark. They just treat it as a
+normal character and consider it to be the first character in the first cell. The Workflow Manager
+discards the byte-order mark before parsing the CSV.
+Excel
+
+-
+
If you open a CSV file in Microsoft Excel and the text is garbled, you should open the file
+ in a text editor that supports UTF-8 and see if the text is garbled there too.
+
+-
+
When saving a CSV file from Excel, if you select "CSV (Comma delimited)(*.csv)", Excel will
+ silently replace East Asian characters with question marks. Selecting
+ "CSV UTF-8 (Comma delimited) (.csv)" preserves the East Asian characters, but it adds a
+ byte-order mark to the file.
+
+-
+
If you open a UTF-8-encoded file in Excel, it will treat it as ISO-8859-1 unless the file has
+ a UTF-8 byte-order mark.
+
+
+Using a byte-order mark with UTF-8 is uncommon because the UTF-8 encoding does not have endianess
+like UTF-16 and UTF-32. The byte-order mark added by Excel can cause problems because a
+lot of software does not expect it to be present.
+As an example, consider an Excel spreadsheet with the following content:
+
+
+
+| Col,1 |
+Col,2 |
+
+
+
+
+| item1 |
+item2 |
+
+
+
+If you save that as "CSV UTF-8 (Comma delimited) (.csv)" and then cat the file you will get:
+"Col,1","Col,2"
+item1,item2
+
+Since the two column names contain commas, the cells need to be escaped with quotation marks so that
+the internal comma is not interpreted as a record separator. If you parse that file with
+Python's built-in CSV parser you get the following
+result:
+
+
+
+| "Col |
+1" |
+Col,2 |
+
+
+
+
+| item1 |
+item2 |
+ |
+
+
+
+While the first cell above appears to be four characters in length, it is actually five. The first
+character is U+FEFF (ZERO WIDTH NO-BREAK SPACE). Since the first cell starts with
+U+FEFF (ZERO WIDTH NO-BREAK SPACE), rather than U+0022 (QUOTATION MARK), the first comma is
+unescaped so it is interpreted as the record separator.
+Since the byte-order mark is invisible when rendered, simply printing the content will not reveal
+the issue. It is visible in a hexdump produced with hexdump -C on Linux or Format-Hex in
+Windows Powershell. In the output from hexdump -C below, you can see that there is content before
+the quotation mark and that content matches the UTF-8-encoded byte-order mark of 0xEF, 0xBB,
+0xBF.
+00000000 ef bb bf 22 43 6f 6c 2c 31 22 2c 22 43 6f 6c 2c |..."Col,1","Col,|
+00000010 32 22 0d 0a 69 74 65 6d 31 2c 69 74 65 6d 32 0d |2"..item1,item2.|
+00000020 0a |.|
+
+CSV_COLS Output File
+When media selectors are used, a copy of the input file with the specified sections replaced by
+component output is produced. The URI to the file will be present in the
+$.media.*.mediaSelectorsOutputUri field.
+Below is an example of a job that uses CSV_COLS media selectors. The job uses a two-stage
+pipeline. The first stage performs language identification. The second performs translation.
+{
+ "algorithmProperties": {},
+ "buildOutput": true,
+ "jobProperties": {},
+ "media": [
+ {
+ "mediaUri": "file:///opt/mpf/share/remote-media/test-csv-translation.csv",
+ "properties": {},
+ "mediaSelectorsOutputAction": "ARGOS TRANSLATION (WITH FF REGION AND NO TASK MERGING) ACTION",
+ "mediaSelectors": [
+ {
+ "type": "CSV_COLS",
+ "expression": "Spanish",
+ "resultDetectionProperty": "TRANSLATION",
+ "selectionProperties": {}
+ },
+ {
+ "type": "CSV_COLS",
+ "expression": "Chinese",
+ "resultDetectionProperty": "TRANSLATION",
+ "selectionProperties": {}
+ }
+ ]
+ }
+ ],
+ "pipelineName": "ARGOS TRANSLATION (WITH FASTTEXT LANGUAGE ID) TEXT FILE PIPELINE",
+ "priority": 4
+}
+
+The input file, test-csv-translation.csv, contains the content below.
+English,Spanish,Chinese
+"Hello, how are you?","¿Hola, cómo estás?",你好吗？
+Where is the library?,¿Dónde está la biblioteca?,图书馆在哪里？
+What time is it?,¿Qué hora es?,现在是几奌？
+
+The mediaSelectorsOutputUri field from the output object will refer to a document containing the
+content below.
+English,Spanish,Chinese
+"Hello, how are you?","Hello, how are you?",How are you?
+Where is the library?,Where's the library?,Where's the library?
+What time is it?,What time is it?,What time is it?
+
+If MEDIA_SELECTORS_DELIMETER was set to " | Translation: ", the file would contain the content
+below.
+English,Spanish,Chinese
+"Hello, how are you?","¿Hola, cómo estás? | Translation: Hello, how are you?",你好吗？ | Translation: How are you?
+Where is the library?,¿Dónde está la biblioteca? | Translation: Where's the library?,图书馆在哪里？ | Translation: Where's the library?
+What time is it?,¿Qué hora es? | Translation: What time is it?,现在是几奌？ | Translation: What time is it?
diff --git a/docs/site/index.html b/docs/site/index.html
index 28e22a0867c2..f0a11030e5e0 100644
--- a/docs/site/index.html
+++ b/docs/site/index.html
@@ -408,5 +408,5 @@ Overview
diff --git a/docs/site/search/search_index.json b/docs/site/search/search_index.json
index f781857d289d..1fdee4e561c5 100644
--- a/docs/site/search/search_index.json
+++ b/docs/site/search/search_index.json
@@ -672,7 +672,7 @@
},
{
"location": "/Media-Selectors-Guide/index.html",
- "text": "NOTICE:\n This software (or technical data) was produced for the U.S. Government under contract,\nand is subject to the Rights in Data-General Clause 52.227-14, Alt. IV (DEC 2007). Copyright 2025\nThe MITRE Corporation. All Rights Reserved.\n\n\nMedia Selectors Overview\n\n\nMedia selectors allow users to specify that only specific sections of a document should be\nprocessed. A copy of the input file with the specified sections replaced by component output is\nproduced.\n\n\nNew Job Request Fields\n\n\nBelow is an example of a job that uses media selectors. The job uses a two-stage pipeline.\nThe first stage performs language identification. The second performs translation.\n\n\n{\n \"algorithmProperties\": {},\n \"buildOutput\": true,\n \"jobProperties\": {},\n \"media\": [\n {\n \"mediaUri\": \"file:///opt/mpf/share/remote-media/test-json-path-translation.json\",\n \"properties\": {},\n \"mediaSelectorsOutputAction\": \"ARGOS TRANSLATION (WITH FF REGION AND NO TASK MERGING) ACTION\",\n \"mediaSelectors\": [\n {\n \"type\": \"JSON_PATH\",\n \"expression\": \"$.spanishMessages.*.content\",\n \"resultDetectionProperty\": \"TRANSLATION\",\n \"selectionProperties\": {}\n },\n {\n \"type\": \"JSON_PATH\",\n \"expression\": \"$.chineseMessages.*.content\",\n \"resultDetectionProperty\": \"TRANSLATION\",\n \"selectionProperties\": {}\n }\n ]\n }\n ],\n \"pipelineName\": \"ARGOS TRANSLATION (WITH FASTTEXT LANGUAGE ID) TEXT FILE PIPELINE\",\n \"priority\": 4\n}\n\n\n\n\n\n$.media.*.mediaSelectorsOutputAction\n: Name of the action that produces content for the media\n selectors output file. In the above example, we specify that we want the translated content\n from Argos in the media selectors output file rather than the detected language from the first\n stage.\n\n\n$.media.*.mediaSelectors\n: List of media selectors that will be used for the media.\n\n\n$.media.*.mediaSelectors.*.type\n: The name of the \ntype of media selector\n\n that is used in the \nexpression\n field.\n\n\n$.media.*.mediaSelectors.*.expression\n: A string specifying the sections of the document that\n should be processed. The \ntype\n field specifies the syntax of the expression.\n\n\n$.media.*.mediaSelectors.*.resultDetectionProperty\n: A detection property name from tracks\n produced by the \nmediaSelectorsOutputAction\n. The media selectors output document will be\n populated with the content of the specified property.\n\n\n$.media.*.mediaSelectors.*.selectionProperties\n: Job properties that will only be used for\n sub-jobs created for a specific media selector. For example, when performing Argos translation\n on a JSON file in a single-stage pipeline without an upstream language detection stage, this\n could set \nDEFAULT_SOURCE_LANGUAGE=es\n for some media selectors and\n \nDEFAULT_SOURCE_LANGUAGE=zh\n for others.\n\n\n\n\nNew Job Properties\n\n\n\n\nMEDIA_SELECTORS_DELIMETER\n: When not provided and a job uses media selectors, the selected parts\n of the document will be replaced with the action output. When provided, the selected parts of\n the document will contain the original content, followed by the value of this property, and\n finally the action output.\n\n\nMEDIA_SELECTORS_DUPLICATE_POLICY\n: Specifies how to handle the case where a job uses media\n selectors and there are multiple outputs for a single selection. When set to \nLONGEST\n, the\n longer of the two outputs is chosen and the shorter one is discarded. When set to \nERROR\n,\n duplicates are considered an error. When set to \nJOIN\n, the duplicates are combined using\n \n|\n as a delimiter.\n\n\nMEDIA_SELECTORS_NO_MATCHES_IS_ERROR\n: When true and a job uses media selectors, an error will be\n generated when none of the selectors match content from the media.\n\n\n\n\nMedia Selector Types\n\n\nJSON_PATH\n is only type currently supported, but others are planned.\n\n\nJSON_PATH\n\n\nUsed to extract content for JSON files. Uses the \"Jayway JsonPath\" library to parse the expressions.\nThe specific syntax supported is available on their\n\nGitHub page\n. JsonPath\nexpressions are case-sensitive.\n\n\nWhen extracting content from the document, only strings, arrays, and objects are considered. All\nother JSON types are ignored. When the JsonPath expression matches an array, each element is\nrecursively explored. When the expression matches an object, keys are left unchanged and each value\nof the object is recursively explored.\n\n\nJSON_PATH Matching Example\n\n\n{\n \"key1\": [\"a\", \"b\", \"c\"],\n \"key2\": {\n \"key3\": [\n {\n \"key4\": [\"d\", \"e\"],\n \"key5\": [\"f\", \"g\"],\n \"key6\": 6\n }\n ]\n }\n}\n\n\n\n\n\n\n\n\n\nExpression\n\n\nMatches\n\n\n\n\n\n\n\n\n\n\n$\n\n\na, b, c, d, e, f, g\n\n\n\n\n\n\n$.*\n\n\na, b, c, d, e, f, g\n\n\n\n\n\n\n$.key1\n\n\na, b, c\n\n\n\n\n\n\n$.key1[0]\n\n\na\n\n\n\n\n\n\n$.key2\n\n\nd, e, f, g\n\n\n\n\n\n\n$.key2.key3\n\n\nd, e, f, g\n\n\n\n\n\n\n$.key2.key3.*.key4\n\n\nd, e\n\n\n\n\n\n\n$.key2.key3.*.*[0]\n\n\nd, f\n\n\n\n\n\n\n\n\nMedia Selectors Output File\n\n\nWhen media selectors are used, the JsonOutputObject will contain a URI referencing the file\nlocation in the \n$.media.*.mediaSelectorsOutputUri\n field.\n\n\nThe job from the \nNew Job Request Fields section\n could be used with the\ndocument below.\n\n\n{\n \"otherStuffKey\": [\"other stuff value\"],\n \"spanishMessages\": [\n {\n \"to\": \"spanish recipient 1\",\n \"from\": \"spanish sender 1\",\n \"content\": \"\u00bfHola, c\u00f3mo est\u00e1s?\"\n },\n {\n \"to\": \"spanish recipient 2\",\n \"from\": \"spanish sender 2\",\n \"content\": \"\u00bfD\u00f3nde est\u00e1 la biblioteca?\"\n }\n ],\n \"chineseMessages\": [\n {\n \"to\": \"chinese recipient 1\",\n \"from\": \"chinese sender 1\",\n \"content\": \"\u73b0\u5728\u662f\u51e0\u594c\uff1f\"\n },\n {\n \"to\": \"chinese recipient 2\",\n \"from\": \"chinese sender 2\",\n \"content\": \"\u4f60\u53eb\u4ec0\u4e48\u540d\u5b57\uff1f\"\n },\n {\n \"to\": \"chinese recipient 3\",\n \"from\": \"chinese sender 3\",\n \"content\": \"\u4f60\u5728\u54ea\u91cc\uff1f\"\n }\n ]\n}\n\n\n\nThe \nmediaSelectorsOutputUri\n field will refer to a document containing the content below.\n\n\n{\n \"otherStuffKey\": [\"other stuff value\"],\n \"spanishMessages\": [\n {\n \"to\": \"spanish recipient 1\",\n \"from\": \"spanish sender 1\",\n \"content\": \"Hello, how are you?\"\n },\n {\n \"to\": \"spanish recipient 2\",\n \"from\": \"spanish sender 2\",\n \"content\": \"Where is the library?\"\n }\n ],\n \"chineseMessages\": [\n {\n \"to\": \"chinese recipient 1\",\n \"from\": \"chinese sender 1\",\n \"content\": \"What time is it?\"\n },\n {\n \"to\": \"chinese recipient 2\",\n \"from\": \"chinese sender 2\",\n \"content\": \"What is your name?\"\n },\n {\n \"to\": \"chinese recipient 3\",\n \"from\": \"chinese sender 3\",\n \"content\": \"Where are you?\"\n }\n ]\n}\n\n\n\nIf \nMEDIA_SELECTORS_DELIMETER\n was set to \" | Translation: \", the file would contain the content\nbelow.\n\n\n{\n \"otherStuffKey\": [\"other stuff value\"],\n \"spanishMessages\": [\n {\n \"to\": \"spanish recipient 1\",\n \"from\": \"spanish sender 1\",\n \"content\": \"\u00bfHola, c\u00f3mo est\u00e1s? | Translation: Hello, how are you?\"\n },\n {\n \"to\": \"spanish recipient 2\",\n \"from\": \"spanish sender 2\",\n \"content\": \"\u00bfD\u00f3nde est\u00e1 la biblioteca? | Translation: Where is the library?\"\n }\n ],\n \"chineseMessages\": [\n {\n \"to\": \"chinese recipient 1\",\n \"from\": \"chinese sender 1\",\n \"content\": \"\u73b0\u5728\u662f\u51e0\u594c\uff1f | Translation: What time is it?\"\n },\n {\n \"to\": \"chinese recipient 2\",\n \"from\": \"chinese sender 2\",\n \"content\": \"\u4f60\u53eb\u4ec0\u4e48\u540d\u5b57\uff1f | Translation: What is your name?\"\n },\n {\n \"to\": \"chinese recipient 3\",\n \"from\": \"chinese sender 3\",\n \"content\": \"\u4f60\u5728\u54ea\u91cc\uff1f | Translation: Where are you?\"\n }\n ]\n}",
+ "text": "NOTICE:\n This software (or technical data) was produced for the U.S. Government under contract,\nand is subject to the Rights in Data-General Clause 52.227-14, Alt. IV (DEC 2007). Copyright 2025\nThe MITRE Corporation. All Rights Reserved.\n\n\nMedia Selectors Overview\n\n\nMedia selectors allow users to specify that only specific sections of a document should be\nprocessed. A copy of the input file with the specified sections replaced by component output is\nproduced.\n\n\nNew Job Request Fields\n\n\nBelow is an example of a job that uses \nJSON_PATH\n media selectors. The job uses a two-stage\npipeline. The first stage performs language identification. The second performs translation.\n\n\n{\n \"algorithmProperties\": {},\n \"buildOutput\": true,\n \"jobProperties\": {},\n \"media\": [\n {\n \"mediaUri\": \"file:///opt/mpf/share/remote-media/test-json-path-translation.json\",\n \"properties\": {},\n \"mediaSelectorsOutputAction\": \"ARGOS TRANSLATION (WITH FF REGION AND NO TASK MERGING) ACTION\",\n \"mediaSelectors\": [\n {\n \"type\": \"JSON_PATH\",\n \"expression\": \"$.spanishMessages.*.content\",\n \"resultDetectionProperty\": \"TRANSLATION\",\n \"selectionProperties\": {}\n },\n {\n \"type\": \"JSON_PATH\",\n \"expression\": \"$.chineseMessages.*.content\",\n \"resultDetectionProperty\": \"TRANSLATION\",\n \"selectionProperties\": {}\n }\n ]\n }\n ],\n \"pipelineName\": \"ARGOS TRANSLATION (WITH FASTTEXT LANGUAGE ID) TEXT FILE PIPELINE\",\n \"priority\": 4\n}\n\n\n\n\n\n$.media.*.mediaSelectorsOutputAction\n: Name of the action that produces content for the media\n selectors output file. In the above example, we specify that we want the translated content\n from Argos in the media selectors output file rather than the detected language from the first\n stage.\n\n\n$.media.*.mediaSelectors\n: List of media selectors that will be used for the media.\n\n\n$.media.*.mediaSelectors.*.type\n: The name of the \ntype of media selector\n\n that is used in the \nexpression\n field.\n\n\n$.media.*.mediaSelectors.*.expression\n: A string specifying the sections of the document that\n should be processed. The \ntype\n field specifies the syntax of the expression.\n\n\n$.media.*.mediaSelectors.*.resultDetectionProperty\n: A detection property name from tracks\n produced by the \nmediaSelectorsOutputAction\n. The media selectors output document will be\n populated with the content of the specified property.\n\n\n$.media.*.mediaSelectors.*.selectionProperties\n: Job properties that will only be used for\n sub-jobs created for a specific media selector. For example, when performing Argos translation\n on a JSON file in a single-stage pipeline without an upstream language detection stage, this\n could set \nDEFAULT_SOURCE_LANGUAGE=es\n for some media selectors and\n \nDEFAULT_SOURCE_LANGUAGE=zh\n for others.\n\n\n\n\nNew Job Properties\n\n\n\n\n\n\nMEDIA_SELECTORS_DELIMETER\n: When not provided and a job uses media selectors, the selected parts\n of the document will be replaced with the action output. When provided, the selected parts of\n the document will contain the original content, followed by the value of this property, and\n finally the action output.\n\n\n\n\n\n\nMEDIA_SELECTORS_DUPLICATE_POLICY\n: Specifies how to handle the case where a job uses media\n selectors and there are multiple outputs for a single selection. When set to \nLONGEST\n, the\n longer of the two outputs is chosen and the shorter one is discarded. When set to \nERROR\n,\n duplicates are considered an error. When set to \nJOIN\n, the duplicates are combined using\n \n|\n as a delimiter.\n\n\n\n\n\n\nMEDIA_SELECTORS_NO_MATCHES_IS_ERROR\n: When true and a job uses media selectors, an error will be\n generated when none of the selectors match content from the media.\n\n\n\n\n\n\nMedia Selector Types\n\n\nJSON_PATH\n and \nCSV_COLS\n are currently supported.\n\n\nJSON_PATH\n\n\nUsed to extract content for JSON files. Uses the \"Jayway JsonPath\" library to parse the expressions.\nThe specific syntax supported is available on their\n\nGitHub page\n. JsonPath\nexpressions are case-sensitive.\n\n\nWhen extracting content from the document, only strings, arrays, and objects are considered. All\nother JSON types are ignored. When the JsonPath expression matches an array, each element is\nrecursively explored. When the expression matches an object, keys are left unchanged and each value\nof the object is recursively explored.\n\n\nJSON_PATH Matching Example\n\n\n{\n \"key1\": [\"a\", \"b\", \"c\"],\n \"key2\": {\n \"key3\": [\n {\n \"key4\": [\"d\", \"e\"],\n \"key5\": [\"f\", \"g\"],\n \"key6\": 6\n }\n ]\n }\n}\n\n\n\n\n\n\n\n\n\nExpression\n\n\nMatches\n\n\n\n\n\n\n\n\n\n\n$\n\n\na, b, c, d, e, f, g\n\n\n\n\n\n\n$.*\n\n\na, b, c, d, e, f, g\n\n\n\n\n\n\n$.key1\n\n\na, b, c\n\n\n\n\n\n\n$.key1[0]\n\n\na\n\n\n\n\n\n\n$.key2\n\n\nd, e, f, g\n\n\n\n\n\n\n$.key2.key3\n\n\nd, e, f, g\n\n\n\n\n\n\n$.key2.key3.*.key4\n\n\nd, e\n\n\n\n\n\n\n$.key2.key3.*.*[0]\n\n\nd, f\n\n\n\n\n\n\n\n\nJSON_PATH Output File\n\n\nWhen media selectors are used, the JsonOutputObject will contain a URI referencing the file\nlocation in the \n$.media.*.mediaSelectorsOutputUri\n field.\n\n\nFor example, consider that the \nmediaUri\n from the job in the\n\nNew Job Request Fields section\n refers to the document below.\n\n\n{\n \"otherStuffKey\": [\"other stuff value\"],\n \"spanishMessages\": [\n {\n \"to\": \"spanish recipient 1\",\n \"from\": \"spanish sender 1\",\n \"content\": \"\u00bfHola, c\u00f3mo est\u00e1s?\"\n },\n {\n \"to\": \"spanish recipient 2\",\n \"from\": \"spanish sender 2\",\n \"content\": \"\u00bfD\u00f3nde est\u00e1 la biblioteca?\"\n }\n ],\n \"chineseMessages\": [\n {\n \"to\": \"chinese recipient 1\",\n \"from\": \"chinese sender 1\",\n \"content\": \"\u73b0\u5728\u662f\u51e0\u594c\uff1f\"\n },\n {\n \"to\": \"chinese recipient 2\",\n \"from\": \"chinese sender 2\",\n \"content\": \"\u4f60\u53eb\u4ec0\u4e48\u540d\u5b57\uff1f\"\n },\n {\n \"to\": \"chinese recipient 3\",\n \"from\": \"chinese sender 3\",\n \"content\": \"\u4f60\u5728\u54ea\u91cc\uff1f\"\n }\n ]\n}\n\n\n\nThe \nmediaSelectorsOutputUri\n field will refer to a document containing the content below.\n\n\n{\n \"otherStuffKey\": [\"other stuff value\"],\n \"spanishMessages\": [\n {\n \"to\": \"spanish recipient 1\",\n \"from\": \"spanish sender 1\",\n \"content\": \"Hello, how are you?\"\n },\n {\n \"to\": \"spanish recipient 2\",\n \"from\": \"spanish sender 2\",\n \"content\": \"Where is the library?\"\n }\n ],\n \"chineseMessages\": [\n {\n \"to\": \"chinese recipient 1\",\n \"from\": \"chinese sender 1\",\n \"content\": \"What time is it?\"\n },\n {\n \"to\": \"chinese recipient 2\",\n \"from\": \"chinese sender 2\",\n \"content\": \"What is your name?\"\n },\n {\n \"to\": \"chinese recipient 3\",\n \"from\": \"chinese sender 3\",\n \"content\": \"Where are you?\"\n }\n ]\n}\n\n\n\nIf \nMEDIA_SELECTORS_DELIMETER\n was set to \" | Translation: \", the file would contain the content\nbelow.\n\n\n{\n \"otherStuffKey\": [\"other stuff value\"],\n \"spanishMessages\": [\n {\n \"to\": \"spanish recipient 1\",\n \"from\": \"spanish sender 1\",\n \"content\": \"\u00bfHola, c\u00f3mo est\u00e1s? | Translation: Hello, how are you?\"\n },\n {\n \"to\": \"spanish recipient 2\",\n \"from\": \"spanish sender 2\",\n \"content\": \"\u00bfD\u00f3nde est\u00e1 la biblioteca? | Translation: Where is the library?\"\n }\n ],\n \"chineseMessages\": [\n {\n \"to\": \"chinese recipient 1\",\n \"from\": \"chinese sender 1\",\n \"content\": \"\u73b0\u5728\u662f\u51e0\u594c\uff1f | Translation: What time is it?\"\n },\n {\n \"to\": \"chinese recipient 2\",\n \"from\": \"chinese sender 2\",\n \"content\": \"\u4f60\u53eb\u4ec0\u4e48\u540d\u5b57\uff1f | Translation: What is your name?\"\n },\n {\n \"to\": \"chinese recipient 3\",\n \"from\": \"chinese sender 3\",\n \"content\": \"\u4f60\u5728\u54ea\u91cc\uff1f | Translation: Where are you?\"\n }\n ]\n}\n\n\n\nCSV_COLS\n\n\nUsed to extract content from specific columns of a CSV file. The expression itself must be a\nsingle row of CSV listing the columns to extract. The \nCSV_SELECTORS_ARE_INDICES\n job property\ncontrols whether the entries refer to column names or zero-based integer indices.\n\n\nCSV-Specific Job Properties\n\n\n\n\n\n\nCSV_SELECTORS_ARE_INDICES\n: When \nFALSE\n (the default), the selector expression must contain\n column names. When \nTRUE\n the selector should contain the zero-based integer indices of the\n columns that should be processed.\n\n\n\n\n\n\nCSV_CSV_FIRST_ROW_IS_DATA\n: When \nFALSE\n (the default), the first row is considered headers and\n will not be processed. When \nTRUE\n, the first row is considered data and the first row will be\n processed.\n\n\n\n\n\n\nAn issue when processing CSV is that sometimes the first row is considered headers (a.k.a. column\nnames) and in others the first row is actually data and there are no headers.\n\n\n\n\n\n\nIn the default configuration (\nCSV_SELECTORS_ARE_INDICES\n = \nFALSE\n and\n \nCSV_FIRST_ROW_IS_DATA\n = \nFALSE\n), the selector expression refers to column names and the\n first row is not processed as data.\n\n\n\n\n\n\nIf the first row is actual data, and you want to specify the columns by index instead of by\n name in the selector expression, set \nCSV_SELECTORS_ARE_INDICES\n = \nTRUE\n and\n \nCSV_FIRST_ROW_IS_DATA\n = \nTRUE\n.\n\n\n\n\n\n\nIf the first row is headers, and you want to specify the columns by index instead of by\n name in the selector expression, set \nCSV_SELECTORS_ARE_INDICES\n = \nTRUE\n and\n \nCSV_FIRST_ROW_IS_DATA\n = \nFALSE\n.\n\n\n\n\n\n\nCSV_COLS Matching Example\n\n\nThe table below shows combinations of values for \nCSV_SELECTORS_ARE_INDICES\n and\n\nCSV_FIRST_ROW_IS_DATA\n when matched against this CSV content:\n\n\nheader0,header1,\"header,2\"\na,b,c\nd,e,f,g\n\n\n\n\n\n\n\n\n\nExpression\n\n\nCSV_SELECTORS_\nARE_INDICES\n\n\nCSV_FIRST_ROW_\nIS_DATA\n\n\nMatches\n\n\n\n\n\n\n\n\n\n\nheader0,\"header,2\"\n\n\nFALSE\n\n\nFALSE\n\n\na, c, d, f\n\n\n\n\n\n\nheader0,\"header,2\"\n\n\nFALSE\n\n\nTRUE\n\n\nheader0, \"header,2\", a, c, d, f\n\n\n\n\n\n\nheader0,headerX\n\n\nFALSE\n\n\nTRUE / FALSE\n\n\nError\n: \"headerX\" does not exist\n\n\n\n\n\n\nheader0,header,2\n\n\nFALSE\n\n\nTRUE / FALSE\n\n\nError\n: \"header\" and \"2\" do not exist\n\n\n\n\n\n\nheader0,\"header,2\"\n\n\nTRUE\n\n\nTRUE / FALSE\n\n\nError\n: The expression contains non-integers.\n\n\n\n\n\n\n0,2\n\n\nTRUE\n\n\nFALSE\n\n\na, c, d, f\n\n\n\n\n\n\n0,2\n\n\nTRUE\n\n\nTRUE\n\n\nheader0, \"header,2\", a, c, d, f\n\n\n\n\n\n\n0,3,4\n\n\nTRUE\n\n\nFALSE\n\n\na, d, g\n\n\n\n\n\n\n0,2\n\n\nFALSE\n\n\nTRUE / FALSE\n\n\nError\n: There are no columns with \"0\" or \"2\" as the header.\n\n\n\n\n\n\n\n\nCSV Text Encodings\n\n\nWe recommend submitting UTF-8-encoded CSV files, but we do attempt to recognize other text\nencodings. When attempting to determine the input file encoding, the Workflow Manager will inspect\nthe first 12,000 bytes of the file. If all of the 12,000 bytes are valid UTF-8 bytes, then the\nWorkflow Manager will treat the file as UTF-8. Otherwise, the Workflow Manager will use\n\nTika's \nCharsetDetector\n\nto determine the encoding.\n\n\nThe media selectors output file will always be UTF-8-encoded. If the input file was UTF-8-encoded\nand had a byte-order mark, then a byte-order mark will be added to the output file.\n\n\nByte-order mark\n\n\nThe UTF-8, UTF-16, and UTF-32 text encodings may have a byte-order mark present. The byte-order\nmark is the Unicode character named \"ZERO WIDTH NO-BREAK SPACE\" with a code point of U+FEFF. Each\nencoding will encode it as different bytes. For example, in UTF-8 it is encoded with three bytes:\n\n0xEF\n, \n0xBB\n, \n0xBF\n.\n\n\nMany CSV parsers do not have special handling for the byte-order mark. They just treat it as a\nnormal character and consider it to be the first character in the first cell. The Workflow Manager\ndiscards the byte-order mark before parsing the CSV.\n\n\nExcel\n\n\n\n\n\n\nIf you open a CSV file in Microsoft Excel and the text is garbled, you should open the file\n in a text editor that supports UTF-8 and see if the text is garbled there too.\n\n\n\n\n\n\nWhen saving a CSV file from Excel, if you select \"CSV (Comma delimited)(*.csv)\", Excel will\n silently replace East Asian characters with question marks. Selecting\n \"CSV UTF-8 (Comma delimited) (.csv)\" preserves the East Asian characters, but it adds a\n byte-order mark to the file.\n\n\n\n\n\n\nIf you open a UTF-8-encoded file in Excel, it will treat it as ISO-8859-1 unless the file has\n a UTF-8 byte-order mark.\n\n\n\n\n\n\nUsing a byte-order mark with UTF-8 is uncommon because the UTF-8 encoding does not have endianess\nlike UTF-16 and UTF-32. The byte-order mark added by Excel can cause problems because a\nlot of software does not expect it to be present.\n\n\nAs an example, consider an Excel spreadsheet with the following content:\n\n\n\n\n\n\n\n\nCol,1\n\n\nCol,2\n\n\n\n\n\n\n\n\n\n\nitem1\n\n\nitem2\n\n\n\n\n\n\n\n\nIf you save that as \"CSV UTF-8 (Comma delimited) (.csv)\" and then \ncat\n the file you will get:\n\n\n\"Col,1\",\"Col,2\"\nitem1,item2\n\n\n\nSince the two column names contain commas, the cells need to be escaped with quotation marks so that\nthe internal comma is not interpreted as a record separator. If you parse that file with\n\nPython's built-in CSV parser\n you get the following\nresult:\n\n\n\n\n\n\n\n\n\ufeff\"Col\n\n\n1\"\n\n\nCol,2\n\n\n\n\n\n\n\n\n\n\nitem1\n\n\nitem2\n\n\n\n\n\n\n\n\n\n\nWhile the first cell above appears to be four characters in length, it is actually five. The first\ncharacter is U+FEFF (ZERO WIDTH NO-BREAK SPACE). Since the first cell starts with\nU+FEFF (ZERO WIDTH NO-BREAK SPACE), rather than U+0022 (QUOTATION MARK), the first comma is\nunescaped so it is interpreted as the record separator.\n\n\nSince the byte-order mark is invisible when rendered, simply printing the content will not reveal\nthe issue. It is visible in a hexdump produced with \nhexdump -C\n on Linux or \nFormat-Hex\n in\nWindows Powershell. In the output from \nhexdump -C\n below, you can see that there is content before\nthe quotation mark and that content matches the UTF-8-encoded byte-order mark of \n0xEF\n, \n0xBB\n,\n\n0xBF\n.\n\n\n00000000 ef bb bf 22 43 6f 6c 2c 31 22 2c 22 43 6f 6c 2c |...\"Col,1\",\"Col,|\n00000010 32 22 0d 0a 69 74 65 6d 31 2c 69 74 65 6d 32 0d |2\"..item1,item2.|\n00000020 0a |.|\n\n\n\nCSV_COLS Output File\n\n\nWhen media selectors are used, a copy of the input file with the specified sections replaced by\ncomponent output is produced. The URI to the file will be present in the\n\n$.media.*.mediaSelectorsOutputUri\n field.\n\n\nBelow is an example of a job that uses \nCSV_COLS\n media selectors. The job uses a two-stage\npipeline. The first stage performs language identification. The second performs translation.\n\n\n{\n \"algorithmProperties\": {},\n \"buildOutput\": true,\n \"jobProperties\": {},\n \"media\": [\n {\n \"mediaUri\": \"file:///opt/mpf/share/remote-media/test-csv-translation.csv\",\n \"properties\": {},\n \"mediaSelectorsOutputAction\": \"ARGOS TRANSLATION (WITH FF REGION AND NO TASK MERGING) ACTION\",\n \"mediaSelectors\": [\n {\n \"type\": \"CSV_COLS\",\n \"expression\": \"Spanish\",\n \"resultDetectionProperty\": \"TRANSLATION\",\n \"selectionProperties\": {}\n },\n {\n \"type\": \"CSV_COLS\",\n \"expression\": \"Chinese\",\n \"resultDetectionProperty\": \"TRANSLATION\",\n \"selectionProperties\": {}\n }\n ]\n }\n ],\n \"pipelineName\": \"ARGOS TRANSLATION (WITH FASTTEXT LANGUAGE ID) TEXT FILE PIPELINE\",\n \"priority\": 4\n}\n\n\n\nThe input file, \ntest-csv-translation.csv\n, contains the content below.\n\n\nEnglish,Spanish,Chinese\n\"Hello, how are you?\",\"\u00bfHola, c\u00f3mo est\u00e1s?\",\u4f60\u597d\u5417\uff1f\nWhere is the library?,\u00bfD\u00f3nde est\u00e1 la biblioteca?,\u56fe\u4e66\u9986\u5728\u54ea\u91cc\uff1f\nWhat time is it?,\u00bfQu\u00e9 hora es?,\u73b0\u5728\u662f\u51e0\u594c\uff1f\n\n\n\nThe \nmediaSelectorsOutputUri\n field from the output object will refer to a document containing the\ncontent below.\n\n\nEnglish,Spanish,Chinese\n\"Hello, how are you?\",\"Hello, how are you?\",How are you?\nWhere is the library?,Where's the library?,Where's the library?\nWhat time is it?,What time is it?,What time is it?\n\n\n\nIf \nMEDIA_SELECTORS_DELIMETER\n was set to \" | Translation: \", the file would contain the content\nbelow.\n\n\nEnglish,Spanish,Chinese\n\"Hello, how are you?\",\"\u00bfHola, c\u00f3mo est\u00e1s? | Translation: Hello, how are you?\",\u4f60\u597d\u5417\uff1f | Translation: How are you?\nWhere is the library?,\u00bfD\u00f3nde est\u00e1 la biblioteca? | Translation: Where's the library?,\u56fe\u4e66\u9986\u5728\u54ea\u91cc\uff1f | Translation: Where's the library?\nWhat time is it?,\u00bfQu\u00e9 hora es? | Translation: What time is it?,\u73b0\u5728\u662f\u51e0\u594c\uff1f | Translation: What time is it?",
"title": "Media Selectors Guide"
},
{
@@ -682,17 +682,17 @@
},
{
"location": "/Media-Selectors-Guide/index.html#new-job-request-fields",
- "text": "Below is an example of a job that uses media selectors. The job uses a two-stage pipeline.\nThe first stage performs language identification. The second performs translation. {\n \"algorithmProperties\": {},\n \"buildOutput\": true,\n \"jobProperties\": {},\n \"media\": [\n {\n \"mediaUri\": \"file:///opt/mpf/share/remote-media/test-json-path-translation.json\",\n \"properties\": {},\n \"mediaSelectorsOutputAction\": \"ARGOS TRANSLATION (WITH FF REGION AND NO TASK MERGING) ACTION\",\n \"mediaSelectors\": [\n {\n \"type\": \"JSON_PATH\",\n \"expression\": \"$.spanishMessages.*.content\",\n \"resultDetectionProperty\": \"TRANSLATION\",\n \"selectionProperties\": {}\n },\n {\n \"type\": \"JSON_PATH\",\n \"expression\": \"$.chineseMessages.*.content\",\n \"resultDetectionProperty\": \"TRANSLATION\",\n \"selectionProperties\": {}\n }\n ]\n }\n ],\n \"pipelineName\": \"ARGOS TRANSLATION (WITH FASTTEXT LANGUAGE ID) TEXT FILE PIPELINE\",\n \"priority\": 4\n} $.media.*.mediaSelectorsOutputAction : Name of the action that produces content for the media\n selectors output file. In the above example, we specify that we want the translated content\n from Argos in the media selectors output file rather than the detected language from the first\n stage. $.media.*.mediaSelectors : List of media selectors that will be used for the media. $.media.*.mediaSelectors.*.type : The name of the type of media selector \n that is used in the expression field. $.media.*.mediaSelectors.*.expression : A string specifying the sections of the document that\n should be processed. The type field specifies the syntax of the expression. $.media.*.mediaSelectors.*.resultDetectionProperty : A detection property name from tracks\n produced by the mediaSelectorsOutputAction . The media selectors output document will be\n populated with the content of the specified property. $.media.*.mediaSelectors.*.selectionProperties : Job properties that will only be used for\n sub-jobs created for a specific media selector. For example, when performing Argos translation\n on a JSON file in a single-stage pipeline without an upstream language detection stage, this\n could set DEFAULT_SOURCE_LANGUAGE=es for some media selectors and\n DEFAULT_SOURCE_LANGUAGE=zh for others.",
+ "text": "Below is an example of a job that uses JSON_PATH media selectors. The job uses a two-stage\npipeline. The first stage performs language identification. The second performs translation. {\n \"algorithmProperties\": {},\n \"buildOutput\": true,\n \"jobProperties\": {},\n \"media\": [\n {\n \"mediaUri\": \"file:///opt/mpf/share/remote-media/test-json-path-translation.json\",\n \"properties\": {},\n \"mediaSelectorsOutputAction\": \"ARGOS TRANSLATION (WITH FF REGION AND NO TASK MERGING) ACTION\",\n \"mediaSelectors\": [\n {\n \"type\": \"JSON_PATH\",\n \"expression\": \"$.spanishMessages.*.content\",\n \"resultDetectionProperty\": \"TRANSLATION\",\n \"selectionProperties\": {}\n },\n {\n \"type\": \"JSON_PATH\",\n \"expression\": \"$.chineseMessages.*.content\",\n \"resultDetectionProperty\": \"TRANSLATION\",\n \"selectionProperties\": {}\n }\n ]\n }\n ],\n \"pipelineName\": \"ARGOS TRANSLATION (WITH FASTTEXT LANGUAGE ID) TEXT FILE PIPELINE\",\n \"priority\": 4\n} $.media.*.mediaSelectorsOutputAction : Name of the action that produces content for the media\n selectors output file. In the above example, we specify that we want the translated content\n from Argos in the media selectors output file rather than the detected language from the first\n stage. $.media.*.mediaSelectors : List of media selectors that will be used for the media. $.media.*.mediaSelectors.*.type : The name of the type of media selector \n that is used in the expression field. $.media.*.mediaSelectors.*.expression : A string specifying the sections of the document that\n should be processed. The type field specifies the syntax of the expression. $.media.*.mediaSelectors.*.resultDetectionProperty : A detection property name from tracks\n produced by the mediaSelectorsOutputAction . The media selectors output document will be\n populated with the content of the specified property. $.media.*.mediaSelectors.*.selectionProperties : Job properties that will only be used for\n sub-jobs created for a specific media selector. For example, when performing Argos translation\n on a JSON file in a single-stage pipeline without an upstream language detection stage, this\n could set DEFAULT_SOURCE_LANGUAGE=es for some media selectors and\n DEFAULT_SOURCE_LANGUAGE=zh for others.",
"title": "New Job Request Fields"
},
{
"location": "/Media-Selectors-Guide/index.html#new-job-properties",
- "text": "MEDIA_SELECTORS_DELIMETER : When not provided and a job uses media selectors, the selected parts\n of the document will be replaced with the action output. When provided, the selected parts of\n the document will contain the original content, followed by the value of this property, and\n finally the action output. MEDIA_SELECTORS_DUPLICATE_POLICY : Specifies how to handle the case where a job uses media\n selectors and there are multiple outputs for a single selection. When set to LONGEST , the\n longer of the two outputs is chosen and the shorter one is discarded. When set to ERROR ,\n duplicates are considered an error. When set to JOIN , the duplicates are combined using\n | as a delimiter. MEDIA_SELECTORS_NO_MATCHES_IS_ERROR : When true and a job uses media selectors, an error will be\n generated when none of the selectors match content from the media.",
+ "text": "MEDIA_SELECTORS_DELIMETER : When not provided and a job uses media selectors, the selected parts\n of the document will be replaced with the action output. When provided, the selected parts of\n the document will contain the original content, followed by the value of this property, and\n finally the action output. MEDIA_SELECTORS_DUPLICATE_POLICY : Specifies how to handle the case where a job uses media\n selectors and there are multiple outputs for a single selection. When set to LONGEST , the\n longer of the two outputs is chosen and the shorter one is discarded. When set to ERROR ,\n duplicates are considered an error. When set to JOIN , the duplicates are combined using\n | as a delimiter. MEDIA_SELECTORS_NO_MATCHES_IS_ERROR : When true and a job uses media selectors, an error will be\n generated when none of the selectors match content from the media.",
"title": "New Job Properties"
},
{
"location": "/Media-Selectors-Guide/index.html#media-selector-types",
- "text": "JSON_PATH is only type currently supported, but others are planned.",
+ "text": "JSON_PATH and CSV_COLS are currently supported.",
"title": "Media Selector Types"
},
{
@@ -706,9 +706,44 @@
"title": "JSON_PATH Matching Example"
},
{
- "location": "/Media-Selectors-Guide/index.html#media-selectors-output-file",
- "text": "When media selectors are used, the JsonOutputObject will contain a URI referencing the file\nlocation in the $.media.*.mediaSelectorsOutputUri field. The job from the New Job Request Fields section could be used with the\ndocument below. {\n \"otherStuffKey\": [\"other stuff value\"],\n \"spanishMessages\": [\n {\n \"to\": \"spanish recipient 1\",\n \"from\": \"spanish sender 1\",\n \"content\": \"\u00bfHola, c\u00f3mo est\u00e1s?\"\n },\n {\n \"to\": \"spanish recipient 2\",\n \"from\": \"spanish sender 2\",\n \"content\": \"\u00bfD\u00f3nde est\u00e1 la biblioteca?\"\n }\n ],\n \"chineseMessages\": [\n {\n \"to\": \"chinese recipient 1\",\n \"from\": \"chinese sender 1\",\n \"content\": \"\u73b0\u5728\u662f\u51e0\u594c\uff1f\"\n },\n {\n \"to\": \"chinese recipient 2\",\n \"from\": \"chinese sender 2\",\n \"content\": \"\u4f60\u53eb\u4ec0\u4e48\u540d\u5b57\uff1f\"\n },\n {\n \"to\": \"chinese recipient 3\",\n \"from\": \"chinese sender 3\",\n \"content\": \"\u4f60\u5728\u54ea\u91cc\uff1f\"\n }\n ]\n} The mediaSelectorsOutputUri field will refer to a document containing the content below. {\n \"otherStuffKey\": [\"other stuff value\"],\n \"spanishMessages\": [\n {\n \"to\": \"spanish recipient 1\",\n \"from\": \"spanish sender 1\",\n \"content\": \"Hello, how are you?\"\n },\n {\n \"to\": \"spanish recipient 2\",\n \"from\": \"spanish sender 2\",\n \"content\": \"Where is the library?\"\n }\n ],\n \"chineseMessages\": [\n {\n \"to\": \"chinese recipient 1\",\n \"from\": \"chinese sender 1\",\n \"content\": \"What time is it?\"\n },\n {\n \"to\": \"chinese recipient 2\",\n \"from\": \"chinese sender 2\",\n \"content\": \"What is your name?\"\n },\n {\n \"to\": \"chinese recipient 3\",\n \"from\": \"chinese sender 3\",\n \"content\": \"Where are you?\"\n }\n ]\n} If MEDIA_SELECTORS_DELIMETER was set to \" | Translation: \", the file would contain the content\nbelow. {\n \"otherStuffKey\": [\"other stuff value\"],\n \"spanishMessages\": [\n {\n \"to\": \"spanish recipient 1\",\n \"from\": \"spanish sender 1\",\n \"content\": \"\u00bfHola, c\u00f3mo est\u00e1s? | Translation: Hello, how are you?\"\n },\n {\n \"to\": \"spanish recipient 2\",\n \"from\": \"spanish sender 2\",\n \"content\": \"\u00bfD\u00f3nde est\u00e1 la biblioteca? | Translation: Where is the library?\"\n }\n ],\n \"chineseMessages\": [\n {\n \"to\": \"chinese recipient 1\",\n \"from\": \"chinese sender 1\",\n \"content\": \"\u73b0\u5728\u662f\u51e0\u594c\uff1f | Translation: What time is it?\"\n },\n {\n \"to\": \"chinese recipient 2\",\n \"from\": \"chinese sender 2\",\n \"content\": \"\u4f60\u53eb\u4ec0\u4e48\u540d\u5b57\uff1f | Translation: What is your name?\"\n },\n {\n \"to\": \"chinese recipient 3\",\n \"from\": \"chinese sender 3\",\n \"content\": \"\u4f60\u5728\u54ea\u91cc\uff1f | Translation: Where are you?\"\n }\n ]\n}",
- "title": "Media Selectors Output File"
+ "location": "/Media-Selectors-Guide/index.html#json_path-output-file",
+ "text": "When media selectors are used, the JsonOutputObject will contain a URI referencing the file\nlocation in the $.media.*.mediaSelectorsOutputUri field. For example, consider that the mediaUri from the job in the New Job Request Fields section refers to the document below. {\n \"otherStuffKey\": [\"other stuff value\"],\n \"spanishMessages\": [\n {\n \"to\": \"spanish recipient 1\",\n \"from\": \"spanish sender 1\",\n \"content\": \"\u00bfHola, c\u00f3mo est\u00e1s?\"\n },\n {\n \"to\": \"spanish recipient 2\",\n \"from\": \"spanish sender 2\",\n \"content\": \"\u00bfD\u00f3nde est\u00e1 la biblioteca?\"\n }\n ],\n \"chineseMessages\": [\n {\n \"to\": \"chinese recipient 1\",\n \"from\": \"chinese sender 1\",\n \"content\": \"\u73b0\u5728\u662f\u51e0\u594c\uff1f\"\n },\n {\n \"to\": \"chinese recipient 2\",\n \"from\": \"chinese sender 2\",\n \"content\": \"\u4f60\u53eb\u4ec0\u4e48\u540d\u5b57\uff1f\"\n },\n {\n \"to\": \"chinese recipient 3\",\n \"from\": \"chinese sender 3\",\n \"content\": \"\u4f60\u5728\u54ea\u91cc\uff1f\"\n }\n ]\n} The mediaSelectorsOutputUri field will refer to a document containing the content below. {\n \"otherStuffKey\": [\"other stuff value\"],\n \"spanishMessages\": [\n {\n \"to\": \"spanish recipient 1\",\n \"from\": \"spanish sender 1\",\n \"content\": \"Hello, how are you?\"\n },\n {\n \"to\": \"spanish recipient 2\",\n \"from\": \"spanish sender 2\",\n \"content\": \"Where is the library?\"\n }\n ],\n \"chineseMessages\": [\n {\n \"to\": \"chinese recipient 1\",\n \"from\": \"chinese sender 1\",\n \"content\": \"What time is it?\"\n },\n {\n \"to\": \"chinese recipient 2\",\n \"from\": \"chinese sender 2\",\n \"content\": \"What is your name?\"\n },\n {\n \"to\": \"chinese recipient 3\",\n \"from\": \"chinese sender 3\",\n \"content\": \"Where are you?\"\n }\n ]\n} If MEDIA_SELECTORS_DELIMETER was set to \" | Translation: \", the file would contain the content\nbelow. {\n \"otherStuffKey\": [\"other stuff value\"],\n \"spanishMessages\": [\n {\n \"to\": \"spanish recipient 1\",\n \"from\": \"spanish sender 1\",\n \"content\": \"\u00bfHola, c\u00f3mo est\u00e1s? | Translation: Hello, how are you?\"\n },\n {\n \"to\": \"spanish recipient 2\",\n \"from\": \"spanish sender 2\",\n \"content\": \"\u00bfD\u00f3nde est\u00e1 la biblioteca? | Translation: Where is the library?\"\n }\n ],\n \"chineseMessages\": [\n {\n \"to\": \"chinese recipient 1\",\n \"from\": \"chinese sender 1\",\n \"content\": \"\u73b0\u5728\u662f\u51e0\u594c\uff1f | Translation: What time is it?\"\n },\n {\n \"to\": \"chinese recipient 2\",\n \"from\": \"chinese sender 2\",\n \"content\": \"\u4f60\u53eb\u4ec0\u4e48\u540d\u5b57\uff1f | Translation: What is your name?\"\n },\n {\n \"to\": \"chinese recipient 3\",\n \"from\": \"chinese sender 3\",\n \"content\": \"\u4f60\u5728\u54ea\u91cc\uff1f | Translation: Where are you?\"\n }\n ]\n}",
+ "title": "JSON_PATH Output File"
+ },
+ {
+ "location": "/Media-Selectors-Guide/index.html#csv_cols",
+ "text": "Used to extract content from specific columns of a CSV file. The expression itself must be a\nsingle row of CSV listing the columns to extract. The CSV_SELECTORS_ARE_INDICES job property\ncontrols whether the entries refer to column names or zero-based integer indices.",
+ "title": "CSV_COLS"
+ },
+ {
+ "location": "/Media-Selectors-Guide/index.html#csv-specific-job-properties",
+ "text": "CSV_SELECTORS_ARE_INDICES : When FALSE (the default), the selector expression must contain\n column names. When TRUE the selector should contain the zero-based integer indices of the\n columns that should be processed. CSV_CSV_FIRST_ROW_IS_DATA : When FALSE (the default), the first row is considered headers and\n will not be processed. When TRUE , the first row is considered data and the first row will be\n processed. An issue when processing CSV is that sometimes the first row is considered headers (a.k.a. column\nnames) and in others the first row is actually data and there are no headers. In the default configuration ( CSV_SELECTORS_ARE_INDICES = FALSE and\n CSV_FIRST_ROW_IS_DATA = FALSE ), the selector expression refers to column names and the\n first row is not processed as data. If the first row is actual data, and you want to specify the columns by index instead of by\n name in the selector expression, set CSV_SELECTORS_ARE_INDICES = TRUE and\n CSV_FIRST_ROW_IS_DATA = TRUE . If the first row is headers, and you want to specify the columns by index instead of by\n name in the selector expression, set CSV_SELECTORS_ARE_INDICES = TRUE and\n CSV_FIRST_ROW_IS_DATA = FALSE .",
+ "title": "CSV-Specific Job Properties"
+ },
+ {
+ "location": "/Media-Selectors-Guide/index.html#csv_cols-matching-example",
+ "text": "The table below shows combinations of values for CSV_SELECTORS_ARE_INDICES and CSV_FIRST_ROW_IS_DATA when matched against this CSV content: header0,header1,\"header,2\"\na,b,c\nd,e,f,g Expression CSV_SELECTORS_ ARE_INDICES CSV_FIRST_ROW_ IS_DATA Matches header0,\"header,2\" FALSE FALSE a, c, d, f header0,\"header,2\" FALSE TRUE header0, \"header,2\", a, c, d, f header0,headerX FALSE TRUE / FALSE Error : \"headerX\" does not exist header0,header,2 FALSE TRUE / FALSE Error : \"header\" and \"2\" do not exist header0,\"header,2\" TRUE TRUE / FALSE Error : The expression contains non-integers. 0,2 TRUE FALSE a, c, d, f 0,2 TRUE TRUE header0, \"header,2\", a, c, d, f 0,3,4 TRUE FALSE a, d, g 0,2 FALSE TRUE / FALSE Error : There are no columns with \"0\" or \"2\" as the header.",
+ "title": "CSV_COLS Matching Example"
+ },
+ {
+ "location": "/Media-Selectors-Guide/index.html#csv-text-encodings",
+ "text": "We recommend submitting UTF-8-encoded CSV files, but we do attempt to recognize other text\nencodings. When attempting to determine the input file encoding, the Workflow Manager will inspect\nthe first 12,000 bytes of the file. If all of the 12,000 bytes are valid UTF-8 bytes, then the\nWorkflow Manager will treat the file as UTF-8. Otherwise, the Workflow Manager will use Tika's CharsetDetector \nto determine the encoding. The media selectors output file will always be UTF-8-encoded. If the input file was UTF-8-encoded\nand had a byte-order mark, then a byte-order mark will be added to the output file.",
+ "title": "CSV Text Encodings"
+ },
+ {
+ "location": "/Media-Selectors-Guide/index.html#byte-order-mark",
+ "text": "The UTF-8, UTF-16, and UTF-32 text encodings may have a byte-order mark present. The byte-order\nmark is the Unicode character named \"ZERO WIDTH NO-BREAK SPACE\" with a code point of U+FEFF. Each\nencoding will encode it as different bytes. For example, in UTF-8 it is encoded with three bytes: 0xEF , 0xBB , 0xBF . Many CSV parsers do not have special handling for the byte-order mark. They just treat it as a\nnormal character and consider it to be the first character in the first cell. The Workflow Manager\ndiscards the byte-order mark before parsing the CSV.",
+ "title": "Byte-order mark"
+ },
+ {
+ "location": "/Media-Selectors-Guide/index.html#excel",
+ "text": "If you open a CSV file in Microsoft Excel and the text is garbled, you should open the file\n in a text editor that supports UTF-8 and see if the text is garbled there too. When saving a CSV file from Excel, if you select \"CSV (Comma delimited)(*.csv)\", Excel will\n silently replace East Asian characters with question marks. Selecting\n \"CSV UTF-8 (Comma delimited) (.csv)\" preserves the East Asian characters, but it adds a\n byte-order mark to the file. If you open a UTF-8-encoded file in Excel, it will treat it as ISO-8859-1 unless the file has\n a UTF-8 byte-order mark. Using a byte-order mark with UTF-8 is uncommon because the UTF-8 encoding does not have endianess\nlike UTF-16 and UTF-32. The byte-order mark added by Excel can cause problems because a\nlot of software does not expect it to be present. As an example, consider an Excel spreadsheet with the following content: Col,1 Col,2 item1 item2 If you save that as \"CSV UTF-8 (Comma delimited) (.csv)\" and then cat the file you will get: \"Col,1\",\"Col,2\"\nitem1,item2 Since the two column names contain commas, the cells need to be escaped with quotation marks so that\nthe internal comma is not interpreted as a record separator. If you parse that file with Python's built-in CSV parser you get the following\nresult: \ufeff\"Col 1\" Col,2 item1 item2 While the first cell above appears to be four characters in length, it is actually five. The first\ncharacter is U+FEFF (ZERO WIDTH NO-BREAK SPACE). Since the first cell starts with\nU+FEFF (ZERO WIDTH NO-BREAK SPACE), rather than U+0022 (QUOTATION MARK), the first comma is\nunescaped so it is interpreted as the record separator. Since the byte-order mark is invisible when rendered, simply printing the content will not reveal\nthe issue. It is visible in a hexdump produced with hexdump -C on Linux or Format-Hex in\nWindows Powershell. In the output from hexdump -C below, you can see that there is content before\nthe quotation mark and that content matches the UTF-8-encoded byte-order mark of 0xEF , 0xBB , 0xBF . 00000000 ef bb bf 22 43 6f 6c 2c 31 22 2c 22 43 6f 6c 2c |...\"Col,1\",\"Col,|\n00000010 32 22 0d 0a 69 74 65 6d 31 2c 69 74 65 6d 32 0d |2\"..item1,item2.|\n00000020 0a |.|",
+ "title": "Excel"
+ },
+ {
+ "location": "/Media-Selectors-Guide/index.html#csv_cols-output-file",
+ "text": "When media selectors are used, a copy of the input file with the specified sections replaced by\ncomponent output is produced. The URI to the file will be present in the $.media.*.mediaSelectorsOutputUri field. Below is an example of a job that uses CSV_COLS media selectors. The job uses a two-stage\npipeline. The first stage performs language identification. The second performs translation. {\n \"algorithmProperties\": {},\n \"buildOutput\": true,\n \"jobProperties\": {},\n \"media\": [\n {\n \"mediaUri\": \"file:///opt/mpf/share/remote-media/test-csv-translation.csv\",\n \"properties\": {},\n \"mediaSelectorsOutputAction\": \"ARGOS TRANSLATION (WITH FF REGION AND NO TASK MERGING) ACTION\",\n \"mediaSelectors\": [\n {\n \"type\": \"CSV_COLS\",\n \"expression\": \"Spanish\",\n \"resultDetectionProperty\": \"TRANSLATION\",\n \"selectionProperties\": {}\n },\n {\n \"type\": \"CSV_COLS\",\n \"expression\": \"Chinese\",\n \"resultDetectionProperty\": \"TRANSLATION\",\n \"selectionProperties\": {}\n }\n ]\n }\n ],\n \"pipelineName\": \"ARGOS TRANSLATION (WITH FASTTEXT LANGUAGE ID) TEXT FILE PIPELINE\",\n \"priority\": 4\n} The input file, test-csv-translation.csv , contains the content below. English,Spanish,Chinese\n\"Hello, how are you?\",\"\u00bfHola, c\u00f3mo est\u00e1s?\",\u4f60\u597d\u5417\uff1f\nWhere is the library?,\u00bfD\u00f3nde est\u00e1 la biblioteca?,\u56fe\u4e66\u9986\u5728\u54ea\u91cc\uff1f\nWhat time is it?,\u00bfQu\u00e9 hora es?,\u73b0\u5728\u662f\u51e0\u594c\uff1f The mediaSelectorsOutputUri field from the output object will refer to a document containing the\ncontent below. English,Spanish,Chinese\n\"Hello, how are you?\",\"Hello, how are you?\",How are you?\nWhere is the library?,Where's the library?,Where's the library?\nWhat time is it?,What time is it?,What time is it? If MEDIA_SELECTORS_DELIMETER was set to \" | Translation: \", the file would contain the content\nbelow. English,Spanish,Chinese\n\"Hello, how are you?\",\"\u00bfHola, c\u00f3mo est\u00e1s? | Translation: Hello, how are you?\",\u4f60\u597d\u5417\uff1f | Translation: How are you?\nWhere is the library?,\u00bfD\u00f3nde est\u00e1 la biblioteca? | Translation: Where's the library?,\u56fe\u4e66\u9986\u5728\u54ea\u91cc\uff1f | Translation: Where's the library?\nWhat time is it?,\u00bfQu\u00e9 hora es? | Translation: What time is it?,\u73b0\u5728\u662f\u51e0\u594c\uff1f | Translation: What time is it?",
+ "title": "CSV_COLS Output File"
},
{
"location": "/REST-API/index.html",
diff --git a/docs/site/sitemap.xml b/docs/site/sitemap.xml
index 7951b12c741f..2f4f09b8a261 100644
--- a/docs/site/sitemap.xml
+++ b/docs/site/sitemap.xml
@@ -2,162 +2,162 @@
/index.html
- 2025-09-19
+ 2025-12-18
daily
/Release-Notes/index.html
- 2025-09-19
+ 2025-12-18
daily
/License-And-Distribution/index.html
- 2025-09-19
+ 2025-12-18
daily
/Acknowledgements/index.html
- 2025-09-19
+ 2025-12-18
daily
/Install-Guide/index.html
- 2025-09-19
+ 2025-12-18
daily
/Admin-Guide/index.html
- 2025-09-19
+ 2025-12-18
daily
/User-Guide/index.html
- 2025-09-19
+ 2025-12-18
daily
/OpenID-Connect-Guide/index.html
- 2025-09-19
+ 2025-12-18
daily
/Media-Segmentation-Guide/index.html
- 2025-09-19
+ 2025-12-18
daily
/Feed-Forward-Guide/index.html
- 2025-09-19
+ 2025-12-18
daily
/Derivative-Media-Guide/index.html
- 2025-09-19
+ 2025-12-18
daily
/Object-Storage-Guide/index.html
- 2025-09-19
+ 2025-12-18
daily
/Markup-Guide/index.html
- 2025-09-19
+ 2025-12-18
daily
/TiesDb-Guide/index.html
- 2025-09-19
+ 2025-12-18
daily
/Trigger-Guide/index.html
- 2025-09-19
+ 2025-12-18
daily
/Roll-Up-Guide/index.html
- 2025-09-19
+ 2025-12-18
daily
/Health-Check-Guide/index.html
- 2025-09-19
+ 2025-12-18
daily
/Artifact-Extraction-Guide/index.html
- 2025-09-19
+ 2025-12-18
daily
/Quality-Selection-Guide/index.html
- 2025-09-19
+ 2025-12-18
daily
/Media-Selectors-Guide/index.html
- 2025-09-19
+ 2025-12-18
daily
/REST-API/index.html
- 2025-09-19
+ 2025-12-18
daily
/Component-API-Overview/index.html
- 2025-09-19
+ 2025-12-18
daily
/Component-Descriptor-Reference/index.html
- 2025-09-19
+ 2025-12-18
daily
/CPP-Batch-Component-API/index.html
- 2025-09-19
+ 2025-12-18
daily
/Python-Batch-Component-API/index.html
- 2025-09-19
+ 2025-12-18
daily
/Java-Batch-Component-API/index.html
- 2025-09-19
+ 2025-12-18
daily
/GPU-Support-Guide/index.html
- 2025-09-19
+ 2025-12-18
daily
/Contributor-Guide/index.html
- 2025-09-19
+ 2025-12-18
daily
/Development-Environment-Guide/index.html
- 2025-09-19
+ 2025-12-18
daily
/Node-Guide/index.html
- 2025-09-19
+ 2025-12-18
daily
/Workflow-Manager-Architecture/index.html
- 2025-09-19
+ 2025-12-18
daily
/CPP-Streaming-Component-API/index.html
- 2025-09-19
+ 2025-12-18
daily
\ No newline at end of file