Python: More Flexible Dependency Requirements, especially for Optional Deps

[srilman]

Feature Request / Improvement

I started to use PyIceberg as a dependency for an application that uses a lot of data-science related libraries, but I started to run into issues due to many conflicting sub-dependency versions, particularly:

• PyArrow (Optional)
• S3FS (Optional)
• Boto3 (Optional)
• FsSpec (Required, but its version is tied with s3fs I believe)

I was wondering, is it necessary for all of PyIceberg's dependencies to be fixed to one version, or can we have more flexible options. This would be especially helpful for optional dependencies.

I haven't done a deep-dive into the code to see how these dependencies are used, but it looks to me that the latest version of PyArrow is not required.

Query engine

None

[srilman]

@Fokko Any thoughts?

[Fokko]

Thanks for pining for me @srilman

Most libraries can vary the version, and some are fixed. For example, aiobotocore is linked to boto3. For Arrow, I think you're okay using an older version.