Describe the bug
The current implementation of optimize using transform_up approach, which lacked context awareness regarding ancestor nodes. As a result, running the optimizer rule multiple times will produce nested CooperativeExec wrappers.
To Reproduce
No response
Expected behavior
-
Idempotency: Running the rule multiple times does not produce nested CooperativeExec wrappers (e.g., Coop(Coop(Scan))).
-
Context Awareness: If a subtree is already protected by a CooperativeExec (e.g., from a previous optimization pass or manual plan construction), we should not double-wrap its children.
Additional context
Ballista side currently use additional optimizer rule to remove duplication, but it will be better to solve this issue in DF directly
https://github.com/milenkovicm/datafusion-ballista/blob/10fb25b9836e70cca63e86728fbf4ebee3e257c3/ballista/scheduler/src/state/aqe/optimizer_rule/datafusion_patch.rs#L126
Describe the bug
The current implementation of optimize using transform_up approach, which lacked context awareness regarding ancestor nodes. As a result, running the optimizer rule multiple times will produce nested CooperativeExec wrappers.
To Reproduce
No response
Expected behavior
Idempotency: Running the rule multiple times does not produce nested CooperativeExec wrappers (e.g., Coop(Coop(Scan))).
Context Awareness: If a subtree is already protected by a CooperativeExec (e.g., from a previous optimization pass or manual plan construction), we should not double-wrap its children.
Additional context
Ballista side currently use additional optimizer rule to remove duplication, but it will be better to solve this issue in DF directly
https://github.com/milenkovicm/datafusion-ballista/blob/10fb25b9836e70cca63e86728fbf4ebee3e257c3/ballista/scheduler/src/state/aqe/optimizer_rule/datafusion_patch.rs#L126