Skip to content

Python: Use Version Ranges for Various Dependencies#6745

Merged
Fokko merged 1 commit into
apache:masterfrom
srilman:sl-py-version-ranges
Feb 9, 2023
Merged

Python: Use Version Ranges for Various Dependencies#6745
Fokko merged 1 commit into
apache:masterfrom
srilman:sl-py-version-ranges

Conversation

@srilman
Copy link
Copy Markdown
Contributor

@srilman srilman commented Feb 4, 2023

As discussed in #6620, this PR uses version ranges for some dependencies. This will unfix the versions for certain dependencies and allow users to use older versions when install PyIceberg to use as a library. The dependencies changed are:

  • PyArrow from 11.0.0 to 8.0.0-11.0.0. It is an optional dependency used either for FileIO, conversion from Iceberg to Arrow (schema, expression, table, etc.), or a dep for Pandas or DuckDB. Made sure current tests pass with every major version.
  • Pandas from 1.5.3 to 1.4.4-1.5.3. It is only used to read Iceberg tables out to Pandas, which is current done through PyArrow anyways. Tested 1.4.4. PyIceberg used 1.5.2 until recently
  • DuckDB from 0.6.1 to 0.6.0-0.6.1. Similar to Pandas, and PyIceberg used 0.6.0 until recently
  • FsSpec, S3Fs, and ADLSFs from 2023.1.0 to 2022.8.2-2023.1.0. Used for FileIO, and I believe S3Fs and ADLSFs already depend on the same version of FsSpec. Tested with all major versions with success, and PyIceberg has used 2022.8.2, 2022.10.0, and 2022.11.2 in the past
  • Requests from 2.28.2 to 2.28.1-2.28.2 since this is only a patch release and PyIceberg used 2.28.1 until recently.
  • Rich from 13.3.1 to 13.0.0-13.3.1 since this is minor release changes with no significant differences
  • PyYAML from 6.0.0 to 5.4.0-6.0.0 since 5.4 is still used commonly in other tools, and the only change in the major change in the major release was removing Python 2.7 support. Tested with 5.4.0

@github-actions github-actions Bot added the python label Feb 4, 2023
@srilman srilman force-pushed the sl-py-version-ranges branch from e7b9004 to 06f7467 Compare February 6, 2023 14:49
@srilman
Copy link
Copy Markdown
Contributor Author

srilman commented Feb 8, 2023

Ready for review. @Fokko or someone else?

Copy link
Copy Markdown
Contributor

@Fokko Fokko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for pinging me @srilman and sorry for the late review. This looks great, thanks for going over this! 👏🏻

Comment thread python/pyproject.toml
thrift = { version = "0.16.0", optional = true }

boto3 = {version = "1.24.59", optional = true}
boto3 = { version = "1.24.59", optional = true }
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would be cool if we can get some kind of TOML linter.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants