Dynamic Filter marked as not used

[LiaCastaneda]

Describe the bug

For one user the DynamicFilterPhysicalExpr function is getting marked as unused. I'm openning this issue to discuss why this could be happening .
More context on #18566 (comment)

To Reproduce

No response

Expected behavior

No response

Additional context

No response

[adriangb]

I'll note that if the using the Arc pointers, etc. is too much of an issue or we can't resolve this bug / usage we could always go for an explicit API to register "consumers" of a dynamic expression. But I think that'd be a worse user experience (things need to remember to call it, etc.).

[LiaCastaneda]

👋 @tobixdev the inner struct of DynamicFilterPhysicalExpr gets updated (the ref count increases from 1 to 2) when we call with_new_children during the filter pushdown optimization rule.

I'm trying to understand why with_new_children is not being called for your query 🤔 this might be an edge case I overlooked. The reason why I didn't used the strong count of DynamicFilterPhysicalExpr is precisely because when with_new_children is called for DynamicFilterPhysicalExpr it creates a new DynamicFilterPhysicalExpr but keeps the Arc of the inner

[alamb]

Could it be that https://docs.rs/datafusion/latest/datafusion/physical_plan/trait.ExecutionPlan.html#method.reset_state is being called (or should be called)?

[tschwarzinger]

Thank you for opening the issue @LiaCastaneda ! I am not sure what the problem is but I guess it has something to do with our custom data source as our project is far away from production ready.

For example, we currently do not implement reset state in our custom data source. Maybe that's it.

I think the next step is that I'll create a reproducer. Hopefully, then we can identify and fix the problem. I'll try to do it sometime next week.

Thanks again and I'll be back with a reproducer!

[LiaCastaneda]

I suspect this is happening because you have a custom DataSource, that differently to FileScanConfig (used for parquet) doesn't call reassign_expr_columns inside try_pushdown_filters (code), so with_new_children is never called. We should fix this since it should work out of the box for any DataSource, I will try to think of something...

[adriangb]

Maybe we can say it’s used if either one of the outer or inner Arc have > 1 ref?

[LiaCastaneda]

yep that works, if the dynamic filter is used the consumer will have either a ref >1 in the inner struct (because with_new_children was called and cloned inner) or outer struct (with_new_children was not called), otherwise there is no way for the consumer to see updates on the DF.

I have a repro similar to @tobixdev DataSource here and verified @adriangb suggestion fixes it.

[LiaCastaneda]

(will open a PR shortly)