Fix dependency with deptry

[kevinjqliu]

Resolve dependency errors found by deptry (added in #528). From 27 -> 16 errors

• Add botocore to main dependency (4 errors)
• Move typing_extensions from dev dependency to main dependency (5 errors)
• Add pydantic_core to main dependency (1 error)
• Add numpy to main dependency (1 error)

botocore missing from dependency was the main issue behind #527

The rest of the errors are most likely false-positives for which we can tweak to ignore using deptry options.

• build/release related files (build-module.py, dev/provision.py, mkdocs/gen_doc_stubs.py) (8 errors)
• snappy (2 errors). This might seem like a bug where deptry doesn't recognize snappy and python-snappy are the same
• hive_metastore / fb303 (5 errors). These are in the /vendors/ folder, so outside scope of dependency check
• psycopg2-binary (1 error). Not explicitly used in the codebase, but might need it for SQL connector(?)

Deptry Log

Deptry run before:

(venv) ➜  iceberg-python git:(main) deptry .
Assuming the corresponding module name of package 'python-snappy' is 'python_snappy'. Install the package or configure a package_module_name_map entry to override this behaviour.
Scanning 69 files...


build-module.py:26:12: DEP004 'Cython' imported but declared as a dev dependency
build-module.py:27:5: DEP004 'Cython' imported but declared as a dev dependency
build-module.py:28:5: DEP003 'setuptools' imported but it is a transitive dependency
build-module.py:29:5: DEP003 'setuptools' imported but it is a transitive dependency
dev/provision.py:18:1: DEP004 'pyspark' imported but declared as a dev dependency
dev/provision.py:19:1: DEP004 'pyspark' imported but declared as a dev dependency
mkdocs/gen_doc_stubs.py:20:8: DEP001 'griffe' imported but missing from the dependency definitions
mkdocs/gen_doc_stubs.py:21:8: DEP001 'mkdocs_gen_files' imported but missing from the dependency definitions
vendor/hive_metastore/ThriftHiveMetastore.py:28:8: DEP001 'fb303' imported but missing from the dependency definitions

pyiceberg/avro/codecs/snappy_codec.py:27:12: DEP001 'snappy' imported but missing from the dependency definitions
pyiceberg/catalog/hive.py:33:1: DEP001 'hive_metastore' imported but missing from the dependency definitions
pyiceberg/catalog/hive.py:34:1: DEP001 'hive_metastore' imported but missing from the dependency definitions
pyiceberg/catalog/hive.py:50:1: DEP001 'hive_metastore' imported but missing from the dependency definitions
pyiceberg/catalog/hive.py:51:1: DEP001 'hive_metastore' imported but missing from the dependency definitions
pyiceberg/catalog/rest.py:418:9: DEP003 'botocore' imported but it is a transitive dependency
pyiceberg/catalog/rest.py:419:9: DEP003 'botocore' imported but it is a transitive dependency
pyiceberg/io/fsspec.py:33:1: DEP003 'botocore' imported but it is a transitive dependency
pyiceberg/io/fsspec.py:34:1: DEP003 'botocore' imported but it is a transitive dependency
pyiceberg/io/pyarrow.py:57:8: DEP003 'numpy' imported but it is a transitive dependency
pyiceberg/partitioning.py:41:1: DEP004 'typing_extensions' imported but declared as a dev dependency
pyiceberg/table/__init__.py:48:1: DEP004 'typing_extensions' imported but declared as a dev dependency
pyiceberg/table/metadata.py:33:1: DEP004 'typing_extensions' imported but declared as a dev dependency
pyiceberg/table/refs.py:21:1: DEP004 'typing_extensions' imported but declared as a dev dependency
pyiceberg/table/sorting.py:35:1: DEP004 'typing_extensions' imported but declared as a dev dependency
pyiceberg/types.py:53:1: DEP003 'pydantic_core' imported but it is a transitive dependency
pyproject.toml: DEP002 'python-snappy' defined as a dependency but not used in the codebase
pyproject.toml: DEP002 'psycopg2-binary' defined as a dependency but not used in the codebase
Found 27 dependency issues.

For more information, see the documentation: https://fpgmaas.github.io/deptry/

Deptry run after:

(venv) ➜  iceberg-python git:(kevinjqliu/deptry) deptry . 
Assuming the corresponding module name of package 'python-snappy' is 'python_snappy'. Install the package or configure a package_module_name_map entry to override this behaviour.
Scanning 69 files...

build-module.py:26:12: DEP004 'Cython' imported but declared as a dev dependency
build-module.py:27:5: DEP004 'Cython' imported but declared as a dev dependency
build-module.py:28:5: DEP003 'setuptools' imported but it is a transitive dependency
build-module.py:29:5: DEP003 'setuptools' imported but it is a transitive dependency
dev/provision.py:18:1: DEP004 'pyspark' imported but declared as a dev dependency
dev/provision.py:19:1: DEP004 'pyspark' imported but declared as a dev dependency
mkdocs/gen_doc_stubs.py:20:8: DEP001 'griffe' imported but missing from the dependency definitions
mkdocs/gen_doc_stubs.py:21:8: DEP001 'mkdocs_gen_files' imported but missing from the dependency definitions
pyiceberg/avro/codecs/snappy_codec.py:27:12: DEP001 'snappy' imported but missing from the dependency definitions
pyiceberg/catalog/hive.py:33:1: DEP001 'hive_metastore' imported but missing from the dependency definitions
pyiceberg/catalog/hive.py:34:1: DEP001 'hive_metastore' imported but missing from the dependency definitions
pyiceberg/catalog/hive.py:50:1: DEP001 'hive_metastore' imported but missing from the dependency definitions
pyiceberg/catalog/hive.py:51:1: DEP001 'hive_metastore' imported but missing from the dependency definitions
pyproject.toml: DEP002 'python-snappy' defined as a dependency but not used in the codebase
pyproject.toml: DEP002 'psycopg2-binary' defined as a dependency but not used in the codebase
vendor/hive_metastore/ThriftHiveMetastore.py:28:8: DEP001 'fb303' imported but missing from the dependency definitions
Found 16 dependency issues.

For more information, see the documentation: https://fpgmaas.github.io/deptry/

[Fokko]

Do we want to make these dependencies required? You don't need botocore if you're a GCP user. Also, numpy should be optional if you use s3fs.

[github-actions]

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that's incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

[github-actions]

This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.