Hadoop Ingestion s3 renames takes forever

[alonblum1]

After upgrading druid from 0.17.2 to 25.0.0 I noticed huge performance degradation in Hadoop ingestion.
The MapReduce job takes about 30 minutes, but now after its done the task performs multiple s3 renames which takes 5 hours!

Affected Version

25.0.0

Description

I run a daily Hadoop ingestion job that performs "multi" ingestion from s3 & the datasource itself.
It used to run ~30 mins in Druid 0.17.2, but after upgrading to 25.0.0 it takes 5h30m!
The MapReduce phase still takes ~30 mins, and then I see in the logs multiple messages like this which lasts for 5 hours:

2023-10-24T00:15:46,199 INFO [task-runner-0-priority-0] org.apache.druid.indexer.JobHelper - Attempting rename from [s3n://xl8druid-storage/druid/segments/reach/2023-10-22T00:00:00.000Z_2023-10-23T00:00:00.000Z/2023-10-23T18:41:39.192Z/98/index.zip.0] to [s3n://xl8druid-storage/druid/segments/reach/2023-10-22T00:00:00.000Z_2023-10-23T00:00:00.000Z/2023-10-23T18:41:39.192Z/98/index.zip]
2023-10-24T00:16:16,545 INFO [task-runner-0-priority-0] org.apache.druid.indexer.JobHelper - Attempting rename from [s3n://xl8druid-storage/druid/segments/reach/2023-10-22T00:00:00.000Z_2023-10-23T00:00:00.000Z/2023-10-23T18:41:39.192Z/99/index.zip.0] to [s3n://xl8druid-storage/druid/segments/reach/2023-10-22T00:00:00.000Z_2023-10-23T00:00:00.000Z/2023-10-23T18:41:39.192Z/99/index.zip]

[gianm]

Hi @alonblum1—

Renames were part of Hadoop ingestion in 0.17.2 as well, but what changed was in 0.22.0 with this patch: #11194. It moved the renames to the index_hadoop job, from their prior place in the reducers, to fix a race condition.

Check this comment for the rationale: #11194 (comment)

I am surprised to hear that moving the renames to the index_hadoop job makes them take 5 hours longer! I'm wondering if you can provide a bit more detail? Like, how many renames are happening, and how many reducers do you have in your job?

[viadrew]

I've been having similar problems when upgrading to 0.22.0. I have multiple ingest jobs for different metrics, from just one a MapReduce ingest job normally running for ~30 minutes on an EMR cluster, it takes an additional 4 hours. Just from the logs, the number of renames is 3310

[alonblum1]

Hey @gianm
I have 21 reducers configured and 729 renames happened in the last job, each file is about 900MB, took nearly 4 hours for the renames alone

[github-actions]

This issue has been marked as stale due to 280 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If this issue is still
relevant, please simply write any comment. Even if closed, you can still revive the
issue at any time or discuss it on the dev@druid.apache.org list.
Thank you for your contributions.

[github-actions]

This issue has been closed due to lack of activity. If you think that
is incorrect, or the issue requires additional review, you can revive the issue at
any time.

[maytasm]

@alonblum1 @viadrew We are seeing this issue too. Just curious, did y'all find a solution? Currently, we just revert this commit so that we can upgrade our Druid version. Also, I think we should reopen this as this is still an issue (I think)

[github-actions]

This issue has been marked as stale due to 280 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If this issue is still
relevant, please simply write any comment. Even if closed, you can still revive the
issue at any time or discuss it on the dev@druid.apache.org list.
Thank you for your contributions.

[github-actions]

This issue has been closed due to lack of activity. If you think that
is incorrect, or the issue requires additional review, you can revive the issue at
any time.