WIP: CI PLATFORM TESTING CONDAFORGE...DELETEME#6081
Closed
hjmjohnson wants to merge 22 commits intoInsightSoftwareConsortium:mainfrom
Closed
WIP: CI PLATFORM TESTING CONDAFORGE...DELETEME#6081hjmjohnson wants to merge 22 commits intoInsightSoftwareConsortium:mainfrom
hjmjohnson wants to merge 22 commits intoInsightSoftwareConsortium:mainfrom
Conversation
Produces three conda packages from a single ITK build:
- libitk: runtime shared libraries
- libitk-devel: headers + CMake config (depends on libitk)
- libitk-wrapping: SWIG Python wrapping artifacts
Uses rattler-build staging: build-cache so the build runs once and
the three outputs each run their own install script. ITK is built
against conda-forge system libs via ITK_USE_SYSTEM_* for EXPAT,
FFTW, HDF5, JPEG, PNG, TIFF, ZLIB, EIGEN, DOUBLECONVERSION, and
CASTXML. Full context and opt-in features (ccache, compile flags)
in Utilities/conda-packages/{README,README_Advanced,AUDIT}.md.
Co-Authored-By: Hans Johnson <hans-johnson@uiowa.edu>
itk_module_impl() prepended ITK:: to every ${itk-module}_LIBRARIES
entry lacking "::". For ITK_USE_SYSTEM_*=ON third-party modules,
entries are IMPORTED targets produced by find_package (hdf5-shared,
etc.) that ship under the upstream package's own naming; prepending
ITK:: creates a non-existent target in INTERFACE_LINK_LIBRARIES and
fails CMake's generate step.
Use if(TARGET) + the IMPORTED target property to detect externally-
provided targets and link them verbatim, without the backward-compat
shims. ITK-owned targets retain the existing namespace and shims.
Two bare-target branches of Modules/ThirdParty/HDF5/CMakeLists.txt listed hdf5_hl_cpp without the -shared / -static suffix that the sibling entries and the hdf5::hdf5_hl_cpp-* branches use. find_package does not produce a target by that name, so ITKHDF5Module's link interface referenced a non-existent ITK::hdf5_hl_cpp and CMake's generate step failed. Append the matching suffix.
itk_module_link_dependencies() linked PUBLIC/INTERFACE/PRIVATE deps
on the module's library target but not COMPILE_DEPENDS. Those deps'
headers reached the module only via directory-scope include_directories
in _itk_module_use_recurse(), which reads the legacy ${<dep>_INCLUDE_DIRS}
variable. For third-party modules that have moved to target-based
INTERFACE_INCLUDE_DIRECTORIES (e.g. Eigen3 after 4ef824b), that
variable is empty and compilation fails with "Eigen/Eigenvalues:
No such file or directory".
Also link COMPILE_DEPENDS as PRIVATE (computed as TRANSITIVE_DEPENDS
minus PUBLIC_DEPENDS), letting modern CMake's target property
propagation carry interface includes into the module's sources
without leaking into its INTERFACE_LINK_LIBRARIES.
The install(DIRECTORY ${ITK_STUB_DIR}/ ...) call in
Wrapping/Generators/Python/CMakeLists.txt had no COMPONENT argument,
so CMake filed it under the "Unspecified" component. Conda / packaging
flows that split the install across outputs (e.g. libitk runtime vs.
libitk-wrapping for Python bindings) then captured all the stub-dir
contents — .abi3.so Python extensions, .pyi type stubs, and the
libITK*-6.0.so copies staged for Python runtime — under the runtime
package instead of the wrapping package.
Assign the same component the sibling install() calls use
(${WRAP_ITK_INSTALL_COMPONENT_IDENTIFIER}RuntimeLibraries) so every
wrapping artifact lands in one predictable component.
rattler-build writes to ./output/ by default when --output-dir is not specified. Add it to .gitignore so developers iterating on Utilities/conda-packages/ do not accidentally stage those artifacts.
itkVoxBoCUBImageIO.cxx calls gzopen/gzclose/gzread/gzwrite/gztell/ gzflush directly via itk_zlib.h, but ITKReview's itk-module.cmake declares no ITKZLIB dep. Linux shared-library builds resolve the symbols transitively via undefined-at-link-time semantics; macOS ARM64 rejects that and fails the Python extension link. Add ITKZLIB under PRIVATE_DEPENDS (implementation only — no public header in ITKReview/include uses zlib).
Temporarily add a GitHub Actions workflow (conda-recipe-test.yml) that exercises Utilities/conda-packages/recipe.yaml via rattler-build on ubuntu-22.04, windows-2022, and macos-15 matrix runners. Upload the three produced .conda files per OS as artifacts for inspection. Disable the existing ITK.Pixi and ITK.Arm64 workflows on this branch by switching their triggers to workflow_dispatch only, so CI runners focus entirely on the conda-recipe validation path. Original push / pull_request trigger blocks are preserved as comments for trivial revert. MUST BE REVERTED before this branch is opened as a ready-for-review PR to upstream.
CastXML 0.7.0 on conda-forge bundles Clang 20 internally, which cannot parse the __acquire_capability__ pack-expansion syntax that libc++ 22 introduced in <mutex>:422. Without this pin, every wrapped ITK class fails during Python XML generation with: attribute '__acquire_capability__' does not support argument pack expansion Remove this pin when CastXML ships against Clang 22+ (or when conda-forge tightens castxml-feedstock's own libcxx upper bound).
github-actions
Bot
added
type:Infrastructure
conda-forge's Windows sandbox path places CMAKE_CURRENT_SOURCE_DIR
at D:\a\_temp\<channel>\bld\rattler-build_libitk_<unix_ts>\work,
which exceeds ITK's default 50-character limit on CMAKE_CURRENT_SOURCE_DIR.
Configure fails with:
CMake Error at CMakeLists.txt:131 (message):
ITK source code directory path length is too long (69 > 50)
Pass -DITK_SKIP_PATH_LENGTH_CHECKS:BOOL=ON in build.bat. The check is
a fossil from MSBuild on pre-Win10 systems; modern Windows runners
(GitHub Actions, conda-forge) have LongPathsEnabled=1.
GitHub Actions ubuntu-22.04 and macos-15 runners have 2-4 cores vs. a typical dev workstation's 16+; the first CI run reached build step 4322/4701 (~92%) before hitting the 120-min job timeout. Bump to 240 min (well within GitHub's 6-hour per-job ceiling) so Linux and macOS can finish the full build + 3-package install pipeline. Part of the temporary CI platform test; reverts with the rest of the WIP CI commit before merge.
Member
|
Hi Hans looks like some interesting work here.
2-cents look. |
Member
Author
Thanks @blowekamp. This is great feedback to help us along. There are a few commits that were quickly put together as we get the prototype plan in place and are likely candidates for cherry-picking into separate PRs. It would be FABULOUS if you could make some comments on individual commits about which ones should be prioritized for working into separate PR's, which ones are off base, etc. I'll look in the SimpleITK repo to see if any gems of knowledge can help us out. I'll investigate why PS: Most of this work is from my graduate student, but my rebasing of his primary work has given me attribution. I need to return @CavRiley as the primary author and me as a co-author. |
Member
Thanks @CavRiley for the work. With the addition of the condo-forge build code here, and the fixing of ITK build and the ITK Python packaging it is a little much for one topic. But it does happen when developing and needs to be broken down. I would recommend forking the condo-forge/libitk-feedstock repo and keeping the build code there. First work on getting the feedstock to work against a local ITK source code, and fix ITK to work in the condo-forge environment against all the third party library. Updating the feedstock to "best practices" is a separate topic. Adding support for ITK python and swig wrapping is quite another topic that needs some understanding of the build and environment requirements, and how to move forward. AI is very good at generating code, but it is poor at knowing if it should actually be generated. |
Member
There are two SimpleITK feedstocks for conda-forge: libsimpleitk and simpleitk. The first builds the C++ SimpleITK libraries as static, and linked against the libitk, and other thridparty shared libraries. These static libraries are the bulk of the compilation time which instantiate all the ITK C++ templates. The second simpleitk, uses the libsimpleitk static libraries and swig to generate the python libraries for each version in an efficient manner. It was a tedious endeavor to get this multi-layer dependencies working. |
Modules/ThirdParty/DoubleConversion/CMakeLists.txt populated
ITKDoubleConversion_LIBRARIES with get_target_property(LOCATION),
which returns the library file path. On Linux that resolves to a
.so which ld accepts; on Windows it resolves to the .dll, which
link.exe cannot consume and which fails with:
%PREFIX%\Library\bin\double-conversion.dll : fatal error LNK1107:
invalid or corrupt file: cannot read at 0x300
Use the double-conversion::double-conversion imported target
directly — CMake resolves it per-platform (Linux .so, Windows .lib
import library) through normal target-property resolution.
Also emit ITKDoubleConversion_EXPORT_CODE_INSTALL/_BUILD so that
downstream consumers of the installed ITKConfig run find_package
(double-conversion) before include(ITKTargets.cmake); otherwise
ITKTargets.cmake's INTERFACE_LINK_LIBRARIES references a target
that does not yet exist in the downstream session.
Core/Modules edits outside the small initial path list (e.g. a Modules/ThirdParty/DoubleConversion change) were silently skipping the cross-platform CI matrix. For a WIP platform-validation workflow we want every push to retrigger all three runners. Reverts with the rest of the WIP CI commit before merge.
Member
Author
|
FYI: This effort was a side project that came out of suggestions for supporting remote modules in a more maintainable (and hopefully simpler) way. Making a link here to that effort InsightSoftwareConsortium/ITKPythonPackage#302 . |
Wrapping/Generators/Python/CMakeLists.txt interpolated
${CMAKE_INSTALL_PREFIX} and ${py_prefix} directly into an
install(CODE "message(WARNING \"...\")") string. On Windows those
variables contain backslash paths; CMake re-parses the generated
cmake_install.cmake with escape-sequence interpretation active and
rejects "\a" inside the path as an invalid C escape:
CMake Error at build/Wrapping/Generators/Python/cmake_install.cmake:36
(message):
Invalid character escape '\\a'.
Run the two paths through file(TO_CMAKE_PATH ...) before
interpolation so the embedded strings use forward slashes. No-op on
Unix; required on Windows / conda-forge / rattler-build sandboxes
where CMAKE_INSTALL_PREFIX lives under paths like D:\\a\\_temp\\....
hjmjohnson
force-pushed
the
add-conda-recipe
branch
from
bd92101 to
90d9e46
Compare
conda-forge packages advance at roughly monthly cadence while ITK's vendored third-party copies advance roughly annually. An unpinned recipe picks up the newest conda-forge version at build time, so an HDF5 2.0 / TBB 2023 / Eigen 3.5 upload silently breaks the next build. Pin each ITK_USE_SYSTEM_*-eligible dep in the build-cache host: block with floor = ITK's vendored version and ceiling = next expected ABI break. Apply matching pins to libitk.run: and libitk-devel.run:; tbb ceiling reflects the oneTBB 2022 stable line pre-dating the 2023 rewrite. Python floor raised to 3.10. Keep the conda-forge zlib / expat metapackages (not the bare libzlib / libexpat); the metapackages ship CMake find_package compatibility glue that the bare libs lack, and dropping them caused find_package(ZLIB) to fall back to the host system libz (1.2.11). Add numpy to host: explicitly. FindPython3 COMPONENTS NumPy needs its headers at configure time; earlier builds picked numpy up transitively, but make the dep explicit so a solver change cannot drop it.
hjmjohnson
force-pushed
the
add-conda-recipe
branch
from
90d9e46 to
13b374e
Compare
Windows cl /DLL produces no .lib import library for a shared library
with no exported symbols. ITKDeprecated ships only a placeholder .cxx
without an ITK_EXPORT-decorated symbol, so the Windows build links
ITKDeprecated-6.0.dll but no ITKDeprecated-6.0.lib, leaving
ITKTargets.cmake referencing a file that does not exist. The
libitk-devel rattler-build test then fails find_package(ITK) with:
The imported target "ITK::ITKDeprecated" references the file
".../Library/lib/ITKDeprecated-6.0.lib" but this file does not exist
Module_ITKDeprecated is optional and default-off; downstream consumers
that need the deprecated compatibility surface can enable it in their
own builds. Dropping it from the conda recipe avoids shipping a broken
import target on Windows while keeping Linux/macOS parity.
Add two sections to README_Advanced.md: 1. "Local build invocation" — shows the correct way to call rattler-build locally (the dev pixi env does not bundle it; use the rattler cache path or pixi global install). 2. "Stale rattler package cache" — explains the hash mismatch error that occurs on repeated local builds and gives the one-line workaround: rm -rf ~/.cache/rattler/cache/pkgs/libitk-* Root cause documented: the staging cache key includes build-environment package hashes, which change as conda-forge updates packages between runs, forcing a new libitk build with a different hash each time.
On Windows, the GDCM-bundled jpeg libraries (gdcmjpeg8/12/16) install
their static .lib archives under the "DebugDevel" CMake component, not
"Development". ITKTargets.cmake references these .lib files, so without
this component the downstream find_package(ITK) test fails:
CMake Error at ITKTargets.cmake:2172:
The imported target "ITK::gdcmjpeg8" references the file
"%PREFIX%/Library/lib/itkgdcmjpeg8-6.0.lib"
but this file does not exist.
Add DebugDevel to both install scripts for cross-platform consistency.
On Linux/macOS the component produces no output (gdcmjpeg static archives
are not exported there), so it is a safe no-op on those platforms.
The libitk-devel test cmake configure used -GNinja without -DCMAKE_BUILD_TYPE=Release. For single-config generators, the cmake --build --config Release flag is ignored, so MSVC compiled main.cxx with /MDd (Debug CRT / _ITERATOR_DEBUG_LEVEL=2) while the installed VNL static libraries were built with /MD (Release). This caused LNK2038 CRT mismatch errors on all 42 link units. Fix: pass -DCMAKE_BUILD_TYPE=Release to cmake configure on Windows so the test links against the same Release libraries that are installed.
Add a cmake_install.cmake invocation for the ThirdParty component to both the Unix shell and Windows batch install scripts for libitk-devel. Third-party headers and config files vendored into the ITK build tree are part of the development package's interface and are required for downstream projects that include ITK headers directly.
ITKPythonPackage drives installation using PythonWheelRuntimeLibraries component names, but the conda libitk-wrapping package builds with WRAP_ITK_INSTALL_COMPONENT_IDENTIFIER=PythonWrapping, producing a different component suffix. Generate a shim cmake_install.cmake in the ITK cmake config directory at conda package install time. The shim maps each PythonWheel* component request to files already installed under the conda prefix and derives the prefix from CMAKE_CURRENT_LIST_FILE so it is fully relocatable. On Windows, the bat script uses a temporary PowerShell file to write the shim, adapting for .pyd extensions and the conda Lib/site-packages layout.
The libitk-wrapping install script searched $PREFIX/lib/cmake/ITK-*/ for the cmake config directory, but that directory is owned by libitk-devel and does not exist during the libitk-wrapping packaging step. The find returned empty and the script aborted. Fix: look for ITKConfig.cmake in the build tree (where it is always present) to determine the ITK version string, then mkdir -p the installed cmake config dir before writing the shim cmake_install.cmake. libitk-devel contributes the remaining cmake config files to the same directory at user-install time; conda permits multiple packages to share a directory.
Member
Author
|
@CavRiley is working on moving this content to feedstocks for conda-forge. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Temporary draft PR used to exercise cross-platform CI (Linux / Windows / macOS)
for the
Utilities/conda-packages/rattler-build recipe.DO NOT MERGE. DO NOT REVIEW. WILL BE CLOSED/DELETED once the CI matrix completes.
What's on the branch
.github/workflows/conda-recipe-test.yml(rattler-build onubuntu-22.04,windows-2022,macos-15)ITK.PixiandITK.Arm64workflows on this branch viaworkflow_dispatch:-only triggersAI assistance
Claude Code assisted with recipe iteration, CMake modernization analysis for ITK-core fixes, and drafting the CI workflow. Every change was built locally in both bundled and system-library modes before pushing.