MINOR: speed up release script#9070
MINOR: speed up release script#9070vvcephei merged 1 commit intoapache:trunkfrom vvcephei:minor-speed-up-release
Conversation
|
Hey all, after waiting at least 10 hours cumulatively for file uploads (so far) for the 2.5.1 release candidates, I've been feeling motivated to try and improve the situation. Currently, it takes about 2 hours to upload the release artifacts, and almost all of that time is just establishing an ssh connection over and over again. Each connection takes about 6 seconds to establish, and then a few milliseconds to transfer the file. I was a little surprised to see this since it looked like the code was supposed to be using sftp in batch mode, but was actually doing a batch of one command each time. I found in #5374 that this was actually intentional, because the full batch of commands was taking too long and making people think the command had hung. I'm proposing to instead expand the toolchain to use I don't know how long the batch upload was taking before, but the rsyncing the compressed artifact is taking me about one minute. Considering the compilation step a little earlier "hangs" with no output for at least two minutes, this seems to be ok. With this change, the release script takes about five minutes to run instead of two hours. WDYT? |
Fixes slow release due to establishing a separate SSH connection per file to copy. Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>
|
Nice improvement! |
|
Agreed! This significantly sped up putting together 2.6.0RC2.
…On Sat, Aug 1, 2020 at 5:35 PM Ismael Juma ***@***.***> wrote:
Nice improvement!
—
You are receiving this because your review was requested.
Reply to this email directly, view it on GitHub
<#9070 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAAUV7GK6Q2PPUNFYOKYYJ3R6SKCBANCNFSM4PGJM2VA>
.
|
…t-for-generated-requests * apache-github/trunk: (148 commits) MINOR: remove NewTopic#NO_PARTITIONS and NewTopic#NO_REPLICATION_FACTOR as they are duplicate to CreateTopicsRequest#NO_NUM_PARTITIONS and CreateTopicsRequest#NO_REPLICATION_FACTOR (apache#9077) MINOR: Remove staticmethod tag to be able to use logger of instance (apache#9086) MINOR: Adjust 'release.py' script to use shell when using gradlewAll and PGP signing, which were required to build the 2.6.0 RCs (apache#9045) MINOR: Update dependencies for Kafka 2.7 (part 1) (apache#9082) MINOR: INFO log4j when request re-join (apache#9068) MINOR: Recommend Java 11 (apache#9080) KAFKA-10306: GlobalThread should fail on InvalidOffsetException (apache#9075) KAFKA-10158: Fix flaky testDescribeUnderReplicatedPartitionsWhenReassignmentIsInProgress (apache#9022) MINOR: code cleanup for `VOut` inconsistent naming (apache#8907) KAFKA-10246 : AbstractProcessorContext topic() throws NPE (apache#9034) KAFKA-10305: Print usage when parsing fails for ConsumerPerformance (apache#9071) MINOR: removed incorrect deprecation annotations (apache#9061) MINOR: speed up release script (apache#9070) MINOR: add task ':streams:testAll' (apache#9073) KAFKA-10301: Do not clear Partition#remoteReplicasMap during partition assignment updates (apache#9065) KAFKA-10268: dynamic config like "--delete-config log.retention.ms" doesn't work (apache#9051) KAFKA-10300 fix flaky core/group_mode_transactions_test.py (apache#9059) MINOR: Publish metrics package in the javadoc (apache#9036) KAFKA-8264: decrease the record size for flaky test KAFKA-5876: Add new exception types for Interactive Queries (apache#8200) ...
4 participants
Fixes slow release due to establishing a
separate SSH connection per file to copy.
Committer Checklist (excluded from commit message)