Support pandas >= 2.x.x

[Zethson]

Dear everyone,

it's super cool to see this project moving to scverse and some activity here!
I just noticed that pytometry has rather strict requirements for Pandas: https://github.com/scverse/pytometry/blob/main/pyproject.toml#L19 . Pandas 2.x.x has been out for a while and I would be happy to see the latest releases being supported.

[mbuttner]

Hi @Zethson
thank you for your request! I think that this is a very good point. I started working on it, but enforcing pandas >= 2.0.0 in the pyproject.toml results in an installation issue, which seems to originate from the requirements in scanpy (see error message below). When I modify the requirement to pandas >= 1.5.3, there is no error message, however, pip "only" installs pandas 1.5.3. Here's what I can do: I can modify the pyproject.toml and remove the pandas < 2.0.0 requirement. I will also test it locally by installing pandas > 2.0.0 manually and check for emerging errors in the package.

Please let me know if I am missing something here or if you have suggestions.

EDIT: I realized that consensusclustering has the same restriction of not supporting pandas v. 2.*.* @burtonrj
I'll open a separate github issue.

---

Error message:

Installing pytometry locally on my MacBook Air (fresh conda environment with python 3.9):

pip install -e .

Output:

Obtaining [...]/pytometry
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... done
Collecting numpy>=1.20.0 (from pytometry==0.1.5)
  Using cached numpy-1.26.4-cp39-cp39-macosx_11_0_arm64.whl.metadata (61 kB)
Collecting numba>=0.57 (from pytometry==0.1.5)
  Using cached numba-0.59.1-cp39-cp39-macosx_11_0_arm64.whl.metadata (2.7 kB)
Collecting pandas>=2.0.0 (from pytometry==0.1.5)
  Using cached pandas-2.2.2-cp39-cp39-macosx_11_0_arm64.whl.metadata (19 kB)
Collecting anndata (from pytometry==0.1.5)
  Using cached anndata-0.10.7-py3-none-any.whl.metadata (6.6 kB)
Collecting scanpy (from pytometry==0.1.5)
  Using cached scanpy-1.10.1-py3-none-any.whl.metadata (8.6 kB)
Collecting scipy (from pytometry==0.1.5)
  Using cached scipy-1.13.0-cp39-cp39-macosx_12_0_arm64.whl.metadata (60 kB)
Collecting seaborn (from pytometry==0.1.5)
  Using cached seaborn-0.13.2-py3-none-any.whl.metadata (5.4 kB)
Collecting matplotlib (from pytometry==0.1.5)
  Using cached matplotlib-3.9.0-cp39-cp39-macosx_11_0_arm64.whl.metadata (11 kB)
Collecting readfcs>=1.1.0 (from pytometry==0.1.5)
  Using cached readfcs-1.1.8-py2.py3-none-any.whl.metadata (1.9 kB)
Collecting flowutils (from pytometry==0.1.5)
  Using cached FlowUtils-1.1.0-cp39-cp39-macosx_11_0_arm64.whl.metadata (3.0 kB)
Collecting consensusclustering (from pytometry==0.1.5)
  Using cached consensusclustering-0.2.1-py3-none-any.whl.metadata (2.1 kB)
Collecting minisom (from pytometry==0.1.5)
  Using cached MiniSom-2.3.2-py3-none-any.whl
Collecting llvmlite<0.43,>=0.42.0dev0 (from numba>=0.57->pytometry==0.1.5)
  Using cached llvmlite-0.42.0-cp39-cp39-macosx_11_0_arm64.whl.metadata (4.8 kB)
Collecting python-dateutil>=2.8.2 (from pandas>=2.0.0->pytometry==0.1.5)
  Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl.metadata (8.4 kB)
Collecting pytz>=2020.1 (from pandas>=2.0.0->pytometry==0.1.5)
  Using cached pytz-2024.1-py2.py3-none-any.whl.metadata (22 kB)
Collecting tzdata>=2022.7 (from pandas>=2.0.0->pytometry==0.1.5)
  Using cached tzdata-2024.1-py2.py3-none-any.whl.metadata (1.4 kB)
Collecting lamin_utils (from readfcs>=1.1.0->pytometry==0.1.5)
  Using cached lamin_utils-0.13.2-py2.py3-none-any.whl.metadata (1.0 kB)
Collecting fcsparser (from readfcs>=1.1.0->pytometry==0.1.5)
  Using cached fcsparser-0.2.8-py3-none-any.whl.metadata (3.6 kB)
Collecting array-api-compat!=1.5,>1.4 (from anndata->pytometry==0.1.5)
  Using cached array_api_compat-1.6-py3-none-any.whl.metadata (1.4 kB)
Collecting exceptiongroup (from anndata->pytometry==0.1.5)
  Using cached exceptiongroup-1.2.1-py3-none-any.whl.metadata (6.6 kB)
Collecting h5py>=3.1 (from anndata->pytometry==0.1.5)
  Using cached h5py-3.11.0-cp39-cp39-macosx_11_0_arm64.whl.metadata (2.5 kB)
Collecting natsort (from anndata->pytometry==0.1.5)
  Using cached natsort-8.4.0-py3-none-any.whl.metadata (21 kB)
Collecting packaging>=20.0 (from anndata->pytometry==0.1.5)
  Using cached packaging-24.0-py3-none-any.whl.metadata (3.2 kB)
Collecting kneed<0.9.0,>=0.8.5 (from consensusclustering->pytometry==0.1.5)
  Using cached kneed-0.8.5-py3-none-any.whl.metadata (5.5 kB)
INFO: pip is looking at multiple versions of consensusclustering to determine which version is compatible with other requirements. This could take a while.
Collecting anndata (from pytometry==0.1.5)
  Using cached anndata-0.10.6-py3-none-any.whl.metadata (6.6 kB)
  Using cached anndata-0.10.5.post1-py3-none-any.whl.metadata (6.5 kB)
  Using cached anndata-0.10.5-py3-none-any.whl.metadata (6.5 kB)
  Using cached anndata-0.10.4-py3-none-any.whl.metadata (6.4 kB)
  Using cached anndata-0.10.3-py3-none-any.whl.metadata (6.4 kB)
  Using cached anndata-0.10.2-py3-none-any.whl.metadata (6.4 kB)
  Using cached anndata-0.10.1-py3-none-any.whl.metadata (6.4 kB)
INFO: pip is still looking at multiple versions of consensusclustering to determine which version is compatible with other requirements. This could take a while.
  Using cached anndata-0.10.0-py3-none-any.whl.metadata (6.4 kB)
  Using cached anndata-0.9.2-py3-none-any.whl.metadata (6.1 kB)
  Using cached anndata-0.9.1-py3-none-any.whl.metadata (4.6 kB)
  Using cached anndata-0.9.0-py3-none-any.whl.metadata (4.6 kB)
  Using cached anndata-0.8.0-py3-none-any.whl.metadata (4.6 kB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.
  Using cached anndata-0.7.8-py3-none-any.whl.metadata (4.2 kB)
Collecting xlrd<2.0 (from anndata->pytometry==0.1.5)
  Using cached xlrd-1.2.0-py2.py3-none-any.whl.metadata (1.3 kB)
Collecting anndata (from pytometry==0.1.5)
  Using cached anndata-0.7.7-py3-none-any.whl.metadata (4.2 kB)
  Using cached anndata-0.7.6-py3-none-any.whl.metadata (4.0 kB)
  Using cached anndata-0.7.5-py3-none-any.whl.metadata (3.9 kB)
  Using cached anndata-0.7.4-py3-none-any.whl.metadata (2.3 kB)
  Using cached anndata-0.7.3-py3-none-any.whl.metadata (2.3 kB)
  Using cached anndata-0.7.2-py3-none-any.whl.metadata (2.3 kB)
  Using cached anndata-0.7.1-py3-none-any.whl.metadata (2.3 kB)
  Using cached anndata-0.7-py3-none-any.whl.metadata (2.3 kB)
  Using cached anndata-0.6.22.post1-py3-none-any.whl.metadata (2.0 kB)
  Using cached anndata-0.6.22-py3-none-any.whl.metadata (2.0 kB)
  Using cached anndata-0.6.21.tar.gz (56 kB)
  Preparing metadata (setup.py) ... done
  Using cached anndata-0.6.20-py3-none-any.whl.metadata (2.0 kB)
  Using cached anndata-0.6.19-py3-none-any.whl.metadata (2.0 kB)
  Using cached anndata-0.6.18-py3-none-any.whl.metadata (2.0 kB)
  Using cached anndata-0.6.17-py3-none-any.whl.metadata (2.0 kB)
  Using cached anndata-0.6.16.tar.gz (53 kB)
  Preparing metadata (setup.py) ... done
  Using cached anndata-0.6.15-py3-none-any.whl.metadata (2.0 kB)
  Using cached anndata-0.6.14-py3-none-any.whl.metadata (2.0 kB)
  Using cached anndata-0.6.13-py3-none-any.whl.metadata (2.0 kB)
  Using cached anndata-0.6.12.tar.gz (52 kB)
  Preparing metadata (setup.py) ... done
  Using cached anndata-0.6.11.tar.gz (52 kB)
  Preparing metadata (setup.py) ... done
  Using cached anndata-0.6.10.tar.gz (51 kB)
  Preparing metadata (setup.py) ... done
  Using cached anndata-0.6.9.tar.gz (51 kB)
  Preparing metadata (setup.py) ... done
  Using cached anndata-0.6.8.tar.gz (50 kB)
  Preparing metadata (setup.py) ... done
  Using cached anndata-0.6.7.tar.gz (50 kB)
  Preparing metadata (setup.py) ... done
Collecting zarr (from anndata->pytometry==0.1.5)
  Using cached zarr-2.18.1-py3-none-any.whl.metadata (5.7 kB)
Collecting anndata (from pytometry==0.1.5)
  Using cached anndata-0.6.6-py3-none-any.whl.metadata (1.9 kB)
  Using cached anndata-0.6.5.tar.gz (47 kB)
  Preparing metadata (setup.py) ... done
  Using cached anndata-0.6.4-py3-none-any.whl.metadata (1.9 kB)
  Using cached anndata-0.6.3-py3-none-any.whl.metadata (1.9 kB)
  Using cached anndata-0.6.2-py3-none-any.whl.metadata (1.9 kB)
  Using cached anndata-0.6.1-py3-none-any.whl.metadata (1.9 kB)
  Using cached anndata-0.6.tar.gz (45 kB)
  Preparing metadata (setup.py) ... done
  Using cached anndata-0.5.10.post2.tar.gz (44 kB)
  Preparing metadata (setup.py) ... done
  Using cached anndata-0.5.10.post1.tar.gz (44 kB)
  Preparing metadata (setup.py) ... done
  Using cached anndata-0.5.10.tar.gz (44 kB)
  Preparing metadata (setup.py) ... done
  Using cached anndata-0.5.9.tar.gz (44 kB)
  Preparing metadata (setup.py) ... done
  Using cached anndata-0.5.8.post1.tar.gz (44 kB)
  Preparing metadata (setup.py) ... done
  Using cached anndata-0.5.8-py3-none-any.whl.metadata (1.7 kB)
  Using cached anndata-0.5.7.tar.gz (44 kB)
  Preparing metadata (setup.py) ... done
  Using cached anndata-0.5.6.tar.gz (43 kB)
  Preparing metadata (setup.py) ... done
  Using cached anndata-0.5.5.tar.gz (42 kB)
  Preparing metadata (setup.py) ... done
  Using cached anndata-0.5.4-py3-none-any.whl.metadata (1.8 kB)
  Using cached anndata-0.5.3.tar.gz (42 kB)
  Preparing metadata (setup.py) ... done
  Using cached anndata-0.5.2.tar.gz (42 kB)
  Preparing metadata (setup.py) ... done
  Using cached anndata-0.5.1-py3-none-any.whl.metadata (1.8 kB)
  Using cached anndata-0.5-py3-none-any.whl.metadata (1.8 kB)
  Using cached anndata-0.4.4-py3-none-any.whl.metadata (2.0 kB)
  Using cached anndata-0.4.3-py3-none-any.whl.metadata (2.0 kB)
  Using cached anndata-0.4.2-py3-none-any.whl.metadata (2.0 kB)
  Using cached anndata-0.4.1.tar.gz (40 kB)
  Preparing metadata (setup.py) ... done
  Using cached anndata-0.4.tar.gz (39 kB)
  Preparing metadata (setup.py) ... done
  Using cached anndata-0.3.4.tar.gz (27 kB)
  Preparing metadata (setup.py) ... done
  Using cached anndata-0.3.3.tar.gz (27 kB)
  Preparing metadata (setup.py) ... done
  Using cached anndata-0.3.2.tar.gz (27 kB)
  Preparing metadata (setup.py) ... done
  Using cached anndata-0.3.1.tar.gz (27 kB)
  Preparing metadata (setup.py) ... done
  Using cached anndata-0.3.0.3.tar.gz (27 kB)
  Preparing metadata (setup.py) ... done
  Using cached anndata-0.3.0.2.tar.gz (26 kB)
  Preparing metadata (setup.py) ... done
  Using cached anndata-0.3.0.1.tar.gz (26 kB)
  Preparing metadata (setup.py) ... done
  Using cached anndata-0.3.tar.gz (26 kB)
  Preparing metadata (setup.py) ... done
  Using cached anndata-0.2.1.tar.gz (23 kB)
  Preparing metadata (setup.py) ... done
  Using cached anndata-0.2.tar.gz (23 kB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [10 lines of output]
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/private/var/folders/m7/s4yvby3n3zvcy3r2gjdth82c0000gn/T/pip-install-hqty5u66/anndata_3aaab3e3ae9147d6a7c72000c2302aa6/setup.py", line 8, in <module>
          with req_path.open() as requirements:
        File "[...]/miniforge3/envs/pytometry_pd2/lib/python3.9/pathlib.py", line 1252, in open
          return io.open(self, mode, buffering, encoding, errors, newline,
        File "[...]/miniforge3/envs/pytometry_pd2/lib/python3.9/pathlib.py", line 1120, in _opener
          return self._accessor.open(self, flags, mode)
      FileNotFoundError: [Errno 2] No such file or directory: 'scanpy.egg-info/requires.txt'
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

[burtonrj]

Ahh yes, I remember coming across this before. I don't think there will be any issues rolling pandas back to v1.5. I'll fix consensusclustering. If I remember correctly, the issue stemmed from the latest version of seaborn requiring pandas v2. I'll just roll back to an older version of seaborn I think.

[Zethson]

@burtonrj uhhm, rolling back to 1.5.x is NOT the solution that I had in mind. We should rather support Pandas 2.x in all of them!

[burtonrj]

Sorry! I got the wrong end of the stick. The consensusclustering package depends on seaborn v0.12 which is where the pandas v2 issues comes from, I'll update the seaborn and pandas versions there.

Now that is has come up though, the consensusclustering package is used for pytometrys implementation of FlowSOM (its not used anywhere else I think). The @saeyslab now offer an open-source python implementation of FlowSOM (https://github.com/saeyslab/FlowSOM_Python). I wonder if its worth switching out the current implementation for this? They're the original authors of FlowSOM.

[mbuttner]

@burtonrj Thanks for pointing this out!
I do think that the implementation of the Saeys Lab would be preferable to use because I would like to avoid parallel implementations and they depend on pytometry in their implementation.
One concern that I have is that the FlowSOM Python implementation depends on muData in addition, such that we would introduce another dependency to pytometry, which goes against our governance decision to create a lightweight package - on the bright side, MuData is part of the scverse so we would stay within the realm of interoperable packages.

[burtonrj]

With how FlowSOM_Python has been implemented, perhaps it would be better to remove FlowSOM from pytometry and suggest FlowSOM_Python separately as part of the end-users tool chain, and maybe it can be introduced into the scverse more officially at a later date?

I think it would just require some documentation and worked examples of how to use pytometry + FlowSOM_Python?

Perhaps worth breaking this out into a separate issue and linking in the authors of FlowSOM_Python directly

[mbuttner]

I think that you have some good points here. I'll open a new issue for this.