Skip to content

Kinesis input format docs#16840

Merged
zachjsh merged 11 commits intoapache:masterfrom
zachjsh:kinesis-input-format-docs
Aug 6, 2024
Merged

Kinesis input format docs#16840
zachjsh merged 11 commits intoapache:masterfrom
zachjsh:kinesis-input-format-docs

Conversation

@zachjsh
Copy link
Copy Markdown
Contributor

@zachjsh zachjsh commented Aug 5, 2024

Description

Adds docs for kinesis input format

This PR has:

  • been self-reviewed.
  • added documentation for new or modified features or behaviors.
  • a release note entry in the PR description.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added or updated version, license, or notice information in licenses.yaml
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • added integration tests.
  • been tested in a test Druid cluster.

@317brian 317brian self-requested a review August 5, 2024 16:29
@vogievetsky vogievetsky added the Needs web console change Backend API changes that would benefit from frontend support in the web console label Aug 5, 2024
Copy link
Copy Markdown
Contributor

@317brian 317brian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving with just some minor nits. Thanks for doing this and making it consistent with the Kafka stuff!

Comment thread docs/ingestion/data-formats.md Outdated
The `kinesis` input format lets you parse the Kinesis metadata fields in addition to the Kinesis payload value contents.
It should only be used when ingesting from Kinesis.

The `kinesis` input format wraps around the payload parsing input format and augments the data it outputs with the kinesis event timestamp, and partition key.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The `kinesis` input format wraps around the payload parsing input format and augments the data it outputs with the kinesis event timestamp, and partition key.
The `kinesis` input format wraps around the payload parsing input format and augments the data it outputs with the Kinesis event timestamp and partition key, the `ApproximateArrivalTimestamp ` and `PartitionKey` fields in the Kinesis record.

Comment thread docs/ingestion/data-formats.md Outdated
Comment thread docs/ingestion/data-formats.md Outdated
Comment thread docs/ingestion/data-formats.md Outdated
Comment thread docs/ingestion/data-formats.md Outdated
Co-authored-by: 317brian <53799971+317brian@users.noreply.github.com>
@zachjsh zachjsh merged commit c324f09 into apache:master Aug 6, 2024
@zachjsh zachjsh deleted the kinesis-input-format-docs branch August 6, 2024 22:53
@kfaraz kfaraz added this to the 31.0.0 milestone Oct 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Area - Documentation Needs web console change Backend API changes that would benefit from frontend support in the web console

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants