Dynamic auto scale Kinesis-Stream ingest tasks

[zhangyue19921010]

Followed PR for #10524.
The core logic is the same, difference lies in the implementation of computeLagStats between Kafka and Kinesis.

Description

This PR implements and documents the autoscaler based on ingest/kinesis/lag/time metrics for kinesis indexing service.
Also only LagBased autoScalerStrategy is supported for now.

---

Key changed/added classes in this PR

• KinesisSupervisor.java
• KinesisSupervisorIOConfig.java

---

This PR has:

• been self-reviewed.
  • using the concurrency checklist (Remove this item if the PR doesn't have any relation to concurrency.)
• added documentation for new or modified features or behaviors.
• added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
• added or updated version, license, or notice information in licenses.yaml
• added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
• added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
• added integration tests.
• been tested in a test Druid cluster.

[zhangyue19921010]

Failed CI jobs are not this PR related. Maybe retry will be passed :)

[suneet-s]

#10691 documents a few known flaky tests. Any ideas on how to make them less flaky will be much appreciated. I've re-triggered the failing tests - just to see if that fixes it

[zhangyue19921010]

#10691 documents a few known flaky tests. Any ideas on how to make them less flaky will be much appreciated. I've re-triggered the failing tests - just to see if that fixes it

Thanks for your help. Sure I will keep an eye on these jobs and try to do some tunning work if I can.

[zhangyue19921010]

This Change has been running in our Dev cluster for 2 weeks and works fine. So that I believe it is ready to be reviewed.

Hi @pjain1 Sorry to bother you. Are you available to help me review this code? Since you have review the original design and may be more familiar with Stream Tasks autoscaler.

Will appreciate it very much if you could lend me a hand :)

[zhangyue19921010]

Also kinesis-index and kinesis-data-format are passed on my laptop.

kinesis-index :

kinesis-data-format :

[suneet-s]

@zhangyue19921010 any learnings from running in your dev cluster over the last few months? I've just started reviewing the change now. But hearing feedback first hand of anything you've noticed is very helpful! Thanks for your patience on this review.

[zhangyue19921010]

Hi @suneet-s Thanks a lot for your attention. As far as I know this patch works fine on our dev/stg cluster and will deploy to PRD cluster soon.

[pjain1]

code changes looks good to me, never used kinesis indexing service so not so sure about the time based lag thing and how it behaves. If someone is using kinesis service, it would be good if they can try it on their cluster.

[techdocsmith]

I added some suggestions as far as language and style the scaleOutThreshold and scaleInThreshold logic is confusing to me.

[techdocsmith]

same comments as for scale out

[clintropolis]

code changes lgtm 👍

@techdocsmith do the doc changes need to be fixed or is it ok to do as a follow-up?

[techdocsmith]

@clintropolis , assuming I didn't break anything w/ my suggested edits, docs changes LGTM.