Merge PyChaste into Chaste develop

[kwabenantim]

Summary

Supports #273

Modifications

• Renames codegen_python3_venv chaste_python3_venv.
• Installs cppwg (for generating wrappers) into chaste_python3_venv in addition to chaste-codegen which is already installed.
• Adds Chaste_ENABLE_PYCHASTE flag to enable configuring and building PyChaste (default is OFF).
• Requires petsc4py package as a build dependency i.e. find_package(PETSc4py REQUIRED).
• Requires extra Development component for Python i.e. find_package(Python3 3.5 COMPONENTS Interpreter Development).
• Requires extra VTK components including vtkRenderingCore, vtkWrappingPythonCore, ...

Additions

• Adds a new pychaste component with the following folder structure:

Chaste/pychaste/
├── CMakeLists.txt
├── dynamic/
│   ├── config.yaml    # Configuration file for wrapper generation
│   ├── templates/     # Custom Python templates for generating wrappers for specific classes
│   ├── typecasters/   # C++/Python type converters for Boost::Ublas, PETSc, and VTK
│   │   └── 3rdparty/  # Type converters from external projects
├── src/
│   ├── cpp/  # PyChaste-specific C++ code
│   │   ├── cell_based/
│   │   │   ├── AttractingPlaneBoundaryCondition.cpp
│   │   │   ├── AttractingPlaneBoundaryCondition.hpp
│   │   │   ├── PythonSimulationModifier.cpp
│   │   │   ├── PythonSimulationModifier.hpp
│   │   │   ├── VtkSceneModifier.cpp
│   │   │   └── VtkSceneModifier.hpp
│   │   └── visualization/
│   │       ├── AbstractPyChasteActorGenerator.cpp
│   │       ├── AbstractPyChasteActorGenerator.hpp
│   │       ├── CellPopulationPyChasteActorGenerator.cpp
│   │       ├── CellPopulationPyChasteActorGenerator.hpp
│   │       ├── VtkScene.cpp
│   │       └── VtkScene.hpp
│   └── py/  # Python package for PyChaste
│       ├── chaste/
│       │   ├── __init__.py
│       │   ├── cell_based/
│       │   │   ├── __init__.py  # All __init__.py files contain imports from compiled wrappers
│       │   ├── core/
│       │   │   ├── __init__.py
│       │   ├── external/  # Third-party JavaScript libraries for visualization
│       │   ├── mesh/
│       │   │   ├── __init__.py
│       │   ├── ode/
│       │   │   ├── __init__.py 
│       │   ├── pde/
│       │   │   ├── __init__.py
│       │   └── visualization/
│       │   │   ├── __init__.py 
│       ├── conda/  # Scripts for building PyChaste with conda dependencies
│       ├── doc/  # PyChaste documentation
│       │   ├── api/  # API docs
│       │   └── tutorial/  # Tutorials in .md and .ipynb
│       └── pyproject.toml
└── test/
    ├── cell_based/
    ├── core/
    ├── data/
    ├── mesh/
    ├── tutorial/
    └── visualization/

Extra Dependencies

• Requires pybind11 library for building wrappers (automatically fetched during configuration).
• Requires pre-installed Clang and CastXML for generating wrappers.
• Requires pre-installed Xvfb for off-screen rendering.
• Requires pre-installed Python packages for configuration and build:
  • mpi4py
  • petsc4py
• Requires additional Python packages for running:
  • matplotlib
  • numpy
  • xvfbwrapper

Tests

• Adds the following tests to the Continuous test pack which can be run via xvfb-run ctest -L pychaste.

cell_based/TestPyCaBasedCellPopulation.py
cell_based/TestPyMeshBasedCellPopulation.py
cell_based/TestPyNodeBasedCellPopulation.py
cell_based/TestPyVertexBasedCellPopulation.py
core/TestPyFileFinder.py
core/TestPyImports.py
core/TestPyOutputFileHandler.py
core/TestPyPetscTools.py
core/TestPyRandomNumberGenerator.py
core/TestPyTimer.py
core/TestPyVersion.py
core/TestPyWrapperChanges.py
mesh/TestPyChastePoint.py
mesh/TestPyPottsMesh.py
tutorial/TestPyCellSortingTutorial.py
tutorial/TestPyImmersedBoundaryTutorial.py
tutorial/TestPyMeshBasedCellSimulationsTutorial.py
tutorial/TestPyNodeBasedCellSimulationsTutorial.py
tutorial/TestPyPottsBasedCellSimulationsTutorial.py
tutorial/TestPyScratchAssayTutorial.py
tutorial/TestPySpheroidTutorial.py
tutorial/TestPyTensileTestTutorial.py
tutorial/TestPyVertexBasedCellSimulationsTutorial.py
visualization/TestPyJupyterNotebookManager.py
visualization/TestVtkSceneWithCaBased.hpp
visualization/TestVtkSceneWithMeshBased.hpp
visualization/TestVtkSceneWithPottsBased.hpp

• Adds pychaste-ubuntu-tests GitHub Actions workflow to build and run all pychaste tests on different Ubuntu versions (currently 22.04 Jammy and 24.04 Noble) with all dependencies installed from Ubuntu apt repositories.
• Adds pychaste-conda-tests GitHub Actions workflow to build and run all pychaste tests on different versions of Python (currently 3.8 to 3.12) with all dependencies installed from conda-forge.

Infrastructure

• Adds a GeneratePyChasteTutorials.py infrastructure script to convert PyChaste tutorial test files to markdown and Jupyter notebook format.
• Adds update-pychaste-tutorials GitHub Actions workflow to update pychaste tutorials on website.

Additional Licenses

• Added licenses for two third-party typecasters in pychaste/dynamic/typecaster/3rdparty:
  • GNU LGPL Version 3
  • BSD 3-Clause
• Added MIT license for three.js
• Updated CheckForCopyrights.py script to recognise additional licenses.

Building and testing on Ubuntu jammy / noble

# Start a docker container if needed (run subsequent steps without "sudo" if using this)
docker run -it --init --name pychaste-test -p 8888:8888 ubuntu:jammy /bin/bash

# Install Chaste dependencies (Use 'https://chaste.github.io/ubuntu noble/' in line 2 if appropriate)
sudo wget -O /usr/share/keyrings/chaste.asc https://chaste.github.io/chaste.asc
echo "deb [signed-by=/usr/share/keyrings/chaste.asc] https://chaste.github.io/ubuntu jammy/" | \
sudo tee -a /etc/apt/sources.list.d/chaste.list

sudo apt-get update
sudo apt-get install -y chaste-dependencies

# Install additional PyChaste dependencies
sudo apt-get install -y \
  castxml \
  clang \
  python3-matplotlib \
  python3-mpi4py \
  python3-notebook \
  python3-numpy \
  python3-petsc4py-real \
  python3-pip \
  python3-vtk7 \  # Change to python3-vtk9 on noble
  python3-xvfbwrapper \
  xvfb
 
# Activate a venv for installing PyChaste (this step can be skipped on Ubuntu jammy)
python3 -m venv --system-site-packages venv
source venv/bin/activate

 # Configure Chaste with PyChaste enabled
(venv)$ mkdir build && cd build
(venv)$ cmake -DChaste_ENABLE_PYCHASTE=ON ..  # assuming Chaste root dir is ..

# Build PyChaste
(venv)$ make -j 4 pychaste

# Install PyChaste
(venv)$ python3 -m pip install pychaste/package

# Run PyChaste tests
(venv)$ xvfb-run ctest -j 4 -L pychaste

# Use PyChaste in a notebook
(venv)$ python3 -m ipykernel install --user --name venv --display-name "Python (pychaste-test)"  # if running in a venv
(venv)$ jupyter notebook --notebook-dir ../pychaste/src/py/doc/tutorial/   # assuming Chaste root dir is ..

# If running in docker, use
(venv)$ export LIBGL_ALWAYS_SOFTWARE=1
(venv)$ jupyter notebook --allow-root --ip 0.0.0.0 --no-browser --notebook-dir ../pychaste/src/py/doc/tutorial/

# Open one of the tutorial notebooks

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 100.00%. Comparing base (b3b2fad) to head (3a42207).
Report is 14 commits behind head on develop.

Additional details and impacted files

@@            Coverage Diff            @@
##           develop      #286   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files         1029      1029           
  Lines        51383     51383           
=========================================
  Hits         51383     51383           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[kwabenantim]

Added summary of changes.

[mirams]

Added licenses for two third-party typecasters in pychaste/dynamic/typecaster/3rdparty:

• GNU LGPL Version 3

I get nervous about LGPL licences, but the first "L" means we can happily use and distribute this as a library without releasing everything as GPL does it?

[kwabenantim]

I get nervous about LGPL licences, but the first "L" means we can happily use and distribute this as a library without releasing everything as GPL does it?

Yes, it looks that way as long as the source is available. We also have CxxTest licensed under LGPL v2.1.

[mirams]

Everything worked for me within a docker (including xvfb-run ctest -j 4 -L pychaste passing 28 tests) up until the point where I tried to run a jupyter notebook tutorial, and got this:

are those path messages important?

[kwabenantim]

Everything worked for me within a docker (including xvfb-run ctest -j 4 -L pychaste passing 28 tests) up until the point where I tried to run a jupyter notebook tutorial, and got this:

Thanks! I'll have a look. Some of the examples may need updating.

[kwabenantim]

I can remove all the wrapper files from Chaste/pychaste/dynamic/wrappers and make them auto-generated whenever Chaste is configured with the Chaste_ENABLE_PYCHASTE switch on.

The tests will still be able to flag when a class is added to (or removed from) cell_based, and throw an error explaining how to fix it in config.yaml.

The tests will no longer be able to detect changes to method signatures since we'll no longer be keeping any reference wrappers. However, this shouldn't be much of a problem - if the wrappers are continually being regenerated, they'll automatically pick up the changes.

Does this sound okay @mirams @MILeach @jmpf ?

[kwabenantim]

Squash merged to avoid polluting develop branch commit history with generated wrappers.