[HUDI-6863] Revert auto-tuning of dedup parallelism

[yihua]

Change Logs

Before this PR, the auto-tuning logic for dedup parallelism dictates the write parallelism so that the user-configured hoodie.upsert.shuffle.parallelism is ignored. This PR reverts #6802 to fix the issue.

Impact

Performance fix

Risk level

low

Documentation Update

N/A

Contributor's checklist

• Read through contributor's guide
• Change Logs and Impact were stated clearly
• Adequate tests were added if applicable
• CI passed

[nsivabalan]

Lets revisit the problems 6802 was tackliing. Main issue it was addressing is, making our shuffle parallelism dynamic and relative to the incoming df's num partitions. So, if someone is running 1000s of pipelines, they don't need to statically set the right value for shuffle parallelism for each of the 1000 pipelines.

can you help me understand whats the issue we are hitting that warrants us to revert it?
also, this would mean that we are going back to old state where we expect users to explicitly configure the shuffle parallelism.
If so, do we have a plan around dynamically choosing the right shuffle partition value depending on incoming batch?

[yihua]

Lets revisit the problems 6802 was tackliing. Main issue it was addressing is, making our shuffle parallelism dynamic and relative to the incoming df's num partitions. So, if someone is running 1000s of pipelines, they don't need to statically set the right value for shuffle parallelism for each of the 1000 pipelines.

can you help me understand whats the issue we are hitting that warrants us to revert it? also, this would mean that we are going back to old state where we expect users to explicitly configure the shuffle parallelism. If so, do we have a plan around dynamically choosing the right shuffle partition value depending on incoming batch?

This PR does not revert the dynamic determination of the shuffle parallelism. The decided target shuffle parallelism is passed in with "int parallelism" through deduplicateRecords. Without the revert, the user loses the ability to override the parallelism through the shuffle parallelism configs because parallelism can be ignored inside this method and the rest of the write DAG uses the new parallelism.

[nsivabalan]

1 minor comments. source code changes looks good.

[yihua]

CI is green.