From 64a8c3d418a0414c9d2c8672752e4179f1008596 Mon Sep 17 00:00:00 2001 From: Jonathan Wei Date: Tue, 7 Jan 2020 17:44:48 -0600 Subject: [PATCH] Update Kinesis resharding information about task failures (#9104) --- docs/development/extensions-core/kinesis-ingestion.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/development/extensions-core/kinesis-ingestion.md b/docs/development/extensions-core/kinesis-ingestion.md index f0452a4fb1be..dd2a6352ca61 100644 --- a/docs/development/extensions-core/kinesis-ingestion.md +++ b/docs/development/extensions-core/kinesis-ingestion.md @@ -443,11 +443,12 @@ Then when submitting a supervisor-spec, set `deaggregate` to true. ## Resharding -When changing the shard count for a Kinesis stream, there will be a window of time around the resharding operation with early shutdown of Kinesis ingestion tasks. -This occurs because the supervisor will update the shard -> task group mappings as shards are closed and fully read, to ensure that tasks are not running +When changing the shard count for a Kinesis stream, there will be a window of time around the resharding operation with early shutdown of Kinesis ingestion tasks and possible task failures. + +The early shutdowns and task failures are expected, and they occur because the supervisor will update the shard -> task group mappings as shards are closed and fully read, to ensure that tasks are not running with an assignment of closed shards that have been fully read and to ensure a balanced distribution of active shards across tasks. -This window with early task shutdowns will conclude when: +This window with early task shutdowns and possible task failures will conclude when: - All closed shards have been fully read and the Kinesis ingestion tasks have published the data from those shards, committing the "closed" state to metadata storage - Any remaining tasks that had inactive shards in the assignment have been shutdown (these tasks would have been created before the closed shards were completely drained)