From f5142b128db3c88c578f1d85c0ed2e3843485225 Mon Sep 17 00:00:00 2001 From: jacobtolar Date: Tue, 2 Nov 2021 00:26:04 -0500 Subject: [PATCH 1/5] Update docs - Kinesis InputFormat ingestion --- .../extensions-core/kinesis-ingestion.md | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/docs/development/extensions-core/kinesis-ingestion.md b/docs/development/extensions-core/kinesis-ingestion.md index cbf2ffe633c3..3dedc4287190 100644 --- a/docs/development/extensions-core/kinesis-ingestion.md +++ b/docs/development/extensions-core/kinesis-ingestion.md @@ -121,7 +121,7 @@ Where the file `supervisor-spec.json` contains a Kinesis supervisor spec: |`type`|The supervisor type; this should always be `kinesis`.|yes| |`spec`|Container object for the supervisor configuration.|yes| |`dataSchema`|The schema that will be used by the Kinesis indexing task during ingestion. See [`dataSchema`](../../ingestion/ingestion-spec.md#dataschema).|yes| -|`ioConfig`|A KinesisSupervisorIOConfig object for configuring Kafka connection and I/O-related settings for the supervisor and indexing task. See [KinesisSupervisorIOConfig](#kinesissupervisorioconfig) below.|yes| +|`ioConfig`|A KinesisSupervisorIOConfig object for configuring Kinesis connection and I/O-related settings for the supervisor and indexing task. See [KinesisSupervisorIOConfig](#kinesissupervisorioconfig) below.|yes| |`tuningConfig`|A KinesisSupervisorTuningConfig object for configuring performance-related settings for the supervisor and indexing tasks. See [KinesisSupervisorTuningConfig](#kinesissupervisortuningconfig) below.|no| @@ -260,13 +260,17 @@ The following example demonstrates a supervisor spec with `lagBased` autoScaler #### Specifying data format Kinesis indexing service supports both [`inputFormat`](../../ingestion/data-formats.md#input-format) and [`parser`](../../ingestion/data-formats.md#parser) to specify the data format. -The `inputFormat` is a new and recommended way to specify the data format for Kinesis indexing service, -but unfortunately, it doesn't support all data formats supported by the legacy `parser`. -(They will be supported in the future.) +Use the `inputFormat` to specify the data format for Kinesis indexing service unless you need a format only supported by the legacy `parser`. -The supported `inputFormat`s include [`csv`](../../ingestion/data-formats.md#csv), -[`delimited`](../../ingestion/data-formats.md#tsv-delimited), [`json`](../../ingestion/data-formats.md#json), [`avro_stream`](../../ingestion/data-formats.md#avro-stream), [`protobuf`](../../ingestion/data-formats.md#protobuf). -You can also read [`thrift`](../extensions-contrib/thrift.md) formats using `parser`. +Supported `inputFormat`s include: +- `csv` +- `delimited` +- `json` +- `avro_stream` +- `avro_ocf` +- `protobuf` + +For more information, see [Data formats](../../ingestion/data-formats.md). You can also read [`thrift`](../extensions-contrib/thrift.md) formats using `parser`. From 8f5da4046b4c973bcbc1abe8a39b5d05ca7693f8 Mon Sep 17 00:00:00 2001 From: jacobtolar Date: Tue, 2 Nov 2021 00:29:09 -0500 Subject: [PATCH 2/5] Add avro_ocf to list of supported Kafka InputFormats --- docs/development/extensions-core/kafka-ingestion.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/development/extensions-core/kafka-ingestion.md b/docs/development/extensions-core/kafka-ingestion.md index ad3c9c041902..700b79671db6 100644 --- a/docs/development/extensions-core/kafka-ingestion.md +++ b/docs/development/extensions-core/kafka-ingestion.md @@ -262,6 +262,7 @@ Supported `inputFormat`s include: - `json` - `kafka` - `avro_stream` +- `avro_ocf` - `protobuf` For more information, see [Data formats](../../ingestion/data-formats.md). You can also read [`thrift`](../extensions-contrib/thrift.md) formats using `parser`. From 492bbe8239c48db84521e988f6af43ba3372afaa Mon Sep 17 00:00:00 2001 From: jacobtolar Date: Thu, 2 Dec 2021 16:25:50 -0600 Subject: [PATCH 3/5] Remove extra whitespace. --- docs/development/extensions-core/kafka-ingestion.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/development/extensions-core/kafka-ingestion.md b/docs/development/extensions-core/kafka-ingestion.md index bdd05c16ddd7..c3b592f37aab 100644 --- a/docs/development/extensions-core/kafka-ingestion.md +++ b/docs/development/extensions-core/kafka-ingestion.md @@ -253,7 +253,6 @@ FROM "wikiticker" kafka.header.environment kafka.key kafka.timestamp development wiki-edit 1636399229823 ``` - For more information, see [`kafka` data format](../../ingestion/data-formats.md#kafka). ## Submit a supervisor spec From 318f6fb39d1a2d78b6251771fed7f5ba69442926 Mon Sep 17 00:00:00 2001 From: jacobtolar Date: Thu, 2 Dec 2021 16:27:07 -0600 Subject: [PATCH 4/5] Update kafka-supervisor-reference.md --- docs/development/extensions-core/kafka-supervisor-reference.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/development/extensions-core/kafka-supervisor-reference.md b/docs/development/extensions-core/kafka-supervisor-reference.md index e03458529c58..b83c4d79c9b1 100644 --- a/docs/development/extensions-core/kafka-supervisor-reference.md +++ b/docs/development/extensions-core/kafka-supervisor-reference.md @@ -166,8 +166,10 @@ Supported `inputFormat`s include: - `json` - `kafka` - `avro_stream` +- `avro_ocf` - `protobuf` + For more information, see [Data formats](../../ingestion/data-formats.md). You can also read [`thrift`](../extensions-contrib/thrift.md) formats using `parser`. From eb6c15e26c35b759ea4fb7ad30df4773079a7bd0 Mon Sep 17 00:00:00 2001 From: jacobtolar Date: Thu, 2 Dec 2021 16:29:07 -0600 Subject: [PATCH 5/5] Delete extra whitespace. --- docs/development/extensions-core/kafka-supervisor-reference.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/development/extensions-core/kafka-supervisor-reference.md b/docs/development/extensions-core/kafka-supervisor-reference.md index b83c4d79c9b1..af6de0354737 100644 --- a/docs/development/extensions-core/kafka-supervisor-reference.md +++ b/docs/development/extensions-core/kafka-supervisor-reference.md @@ -169,7 +169,6 @@ Supported `inputFormat`s include: - `avro_ocf` - `protobuf` - For more information, see [Data formats](../../ingestion/data-formats.md). You can also read [`thrift`](../extensions-contrib/thrift.md) formats using `parser`.