DataFusion weekly project plan (Andrew Lamb) - Nov 6, 2023

[alamb]

The idea of this ticket is to log plans I have each week for DataFusion related items.

This is largely for my own personal organizational needs and I think it will help me remain better focused, but I also figured it might serve as an interesting summary of what is happening in DataFusion this week

Major items I know for this week:

• Several more editing passes for the SIGMOD paper: Write DataFusion paper for (SIGMOD / VLDB / ICDE) #6782 with @ozankabak @Dandandan @yjshen and @JayjeetAtGithub
• Write up some idea of "growing and sustaining the DataFusion community" that I have discussed with @andygrove @thinkharderdev @sunchao @viirya and others
• Assist with cleaning up array functions, and getting good patterns going: Minor: Support nulls in array_replace, avoid a copy #8054 with @jayzhan211 (e.g. Minor: Support nulls in array_replace, avoid a copy #8054)
• Unify BuiltInScalarFunctions with ScalarUDF: [EPIC] Unify Function Interface (remove BuiltInScalarFunction) #8045, get agreement and get first few PRs ready with @2010YOUY01
• Assist with sort based optimization work started in Combine Equivalence and Ordering equivalence to simplify state #8006 (encapsulating more structs, e.g Encapsulate EquivalenceClass into a struct #8034 and reviews) with @ozankabak and @mustafasrepo
• Review any new ticket / bug reports

[2010YOUY01]

This is a good idea, maybe we can also list some tasks that have been identified as valuable but do not yet have someone working on them.

[alamb]

This is a good idea, maybe we can also list some tasks that have been identified as valuable but do not yet have someone working on them.

that would be awesome. THank you for the idea @2010YOUY01

I have always struggled with "who decides what tasks are valuable" but I suppose the answer has always been something like "whoever has the time to make them happen" 🤔 I'll try and write this idea up more coherently

[alamb]

I updated the "unify function implementation plan" with @2010YOUY01 -- #8045 and made an initial PR to make ScalarUDF fields non pub: #8079

[alamb]

It turns out that #8078 is important for an internal project in IOx, so I plan to work on that as well

[alamb]

I am also trying to help @korowa with #8020 as that is also important for us at InfluxData

[alamb]

Next week's plan: #8151