Fix Windows virtual workspace path separator issue#33
Merged
Conversation
…ward slashes on Windows The virtual workspace mapping keys were being created with str(Path(...)) which uses backslashes on Windows. This caused all mapping lookups to fail on Windows because the JSON stored keys with backslashes while the lookup code normalized incoming paths to forward slashes. Changes: - OperaPhenix: Use .as_posix() when building virtual_relative and real_relative paths - ImageXpress: Use .as_posix() when building virtual_relative and real_relative paths This ensures the mapping keys are always stored with forward slashes in the JSON file regardless of platform, matching the normalization done during lookups. Fixes Windows CI failures where orchestrator found 0 wells due to failed file listings.
…n Windows os.rename() fails on Windows when the destination file already exists with WinError 183 'Cannot create a file when that file already exists'. os.replace() is the atomic replacement operation that works on both Unix and Windows platforms - it atomically replaces the destination file if it exists. This fixes the atomic metadata write failures in multiprocessing mode on Windows where multiple processes were trying to update the same metadata file. Changes: - atomic.py: Replace os.rename() with os.replace() in atomic_write_json() Fixes Windows CI error: [WinError 183] Cannot create a file when that file already exists: '.tmpopenhcs_metadata.json0g40hqyz.json' -> 'openhcs_metadata.json'
…pattern_path
The build_dict_pattern_path() function was hardcoding '/' for path splitting,
which failed on Windows where paths use backslashes.
Changed from:
dir_part, filename = base_path.rsplit('/', 1)
return f"{dir_part}/{well_id}_w{dict_key}_{rest}"
To:
path = Path(base_path)
dir_part = path.parent
filename = path.name
return str(dir_part / f"{well_id}_w{dict_key}_{rest}")
This fixes the error:
ValueError: not enough values to unpack (expected 2, got 1)
Fixes Windows CI error in _filter_special_outputs_for_function when building
channel-specific paths for dict pattern special outputs.
Add comprehensive GUI testing for PyQt6 interface across Linux, Windows, and macOS. Features: - Tests on Python 3.11 and 3.12 - Linux: Uses Xvfb for headless Qt testing - Windows: Uses offscreen platform for headless testing - macOS: Uses offscreen platform for headless testing - Uploads test artifacts (logs, screenshots) on failure - Runs on push to main, PRs, and manual dispatch Test configuration: - QT_QPA_PLATFORM=offscreen for headless mode - OPENHCS_CPU_ONLY=true to skip GPU-dependent tests - --maxfail=5 to stop after 5 failures for faster feedback This ensures GUI code works correctly across all platforms and catches cross-platform issues like path separator problems early.
…table names
OMERO table names cannot contain dots except for the .h5 extension.
Previously, output_path.stem only removed the last extension (.json),
leaving intermediate extensions like .roi.zip in the table name.
For example:
- Path: A01_s001_w1_z001_t001_cell_counts_step7.roi.zip.json
- Old: table_name = 'A01_s001_w1_z001_t001_cell_counts_step7.roi.zip'
- New: table_name = 'A01_s001_w1_z001_t001_cell_counts_step7'
This caused OMERO to reject the table creation with:
omero.InternalException: message = null table as argument
Fix: Use output_path.name.split('.')[0] to remove ALL extensions.
Expand test coverage to include Python 3.13 and 3.14 for both Linux and Windows CI. Changes: - integration-tests-focused: Test on Python 3.11, 3.12, 3.13, 3.14 - integration-tests-windows: Test on Python 3.11, 3.12, 3.13, 3.14 This ensures OpenHCS works correctly across all supported Python versions and catches version-specific issues early.
Pin NumPy < 2.2 for Python 3.14 to work around histogram/bincount bug. NumPy 2.2.x has a compatibility issue with Python 3.14 that causes ValueError in scikit-image's threshold_otsu function. Re-enabled Python 3.14 in CI matrix for both Linux and Windows tests.
5 tasks
Reduce CI jobs from 32 to 10 using smart grouping: Group 1: Python boundary tests (4 jobs) - Test Python 3.11 and 3.14 on both Linux and Windows - Validates oldest and newest supported versions - Catches dependency compatibility issues (like NumPy 3.14 bug) Group 2: Backend/microscope coverage (4 jobs) - Test all backend/microscope combos on both OSes with Python 3.12 - Ensures cross-platform compatibility for all backends Group 3: OMERO tests (2 jobs) - Test OMERO on Linux with Python 3.11 and 3.14 - Linux only (OMERO requires Docker) Total: 10 jobs (down from 32) Coverage: All Python versions, both OSes, all backends, all microscopes Created issue #34 to track NumPy version pin removal.
zeroc-ice fails to build from source on Python 3.11+ due to setuptools compatibility issues. Use pre-built wheels from Glencoe Software (OMERO maintainers) instead: - Python 3.11: zeroc-ice 3.6.5 (cp311, manylinux_2_28) - Python 3.14: zeroc-ice 3.6.5 (cp314, manylinux_2_28) This matches the approach used by napari and other OMERO-dependent projects.
Python 3.14 is too new - zeroc-ice pre-built wheels only available up to Python 3.12 (as of 20240202 release). Testing on 3.11 and 3.13 provides good coverage while we wait for OMERO ecosystem to catch up. Total CI jobs: 11 (4 boundary + 4 backend/microscope + 2 OMERO + 1 wheel test)
zeroc-ice pre-built wheels only available up to Python 3.12 (as of 20240202 release). Python 3.13 and 3.14 not yet supported by OMERO ecosystem. Testing on 3.11 (oldest) and 3.12 (newest supported) provides boundary coverage for OMERO integration. Total CI jobs: 11
Python 3.14 is too new - no pre-built wheels available: - NumPy: builds from source, stalls on Windows CI (5+ min timeout) - zeroc-ice (OMERO): no wheels at all Changed boundary testing from 3.11/3.14 to 3.11/3.13 (well-supported versions). Total CI jobs: 11 (4 boundary + 4 backend/microscope + 2 OMERO + 1 wheel) Python 3.14 support will be added when ecosystem catches up (likely Q1 2025).
Added comprehensive header comment documenting: - 11 total jobs organized into 4 groups - Boundary testing strategy (3.11, 3.13) - Python 3.14 exclusion rationale - OMERO limitations (Python 3.11-3.12 only) - Coverage: all Python versions, OSes, backends, microscopes
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Windows CI tests were failing with two separate issues:
Issue 1: Virtual workspace path separator mismatch
The virtual workspace mapping keys were being created with backslashes on Windows, causing "No wells found for processing" errors.
Root Cause: When building the
workspace_mappingdictionary, the code usedstr(Path(...))which converts to backslashes on Windows:'Images\\r01c01f007p001-ch1sk1fk1fl1.tiff''Images/r01c01f007p001-ch1sk1fk1fl1.tiff'The mapping lookup code normalizes incoming paths to forward slashes with
.replace('\\', '/'), but the mapping keys themselves had backslashes on Windows, causing all lookups to fail.Issue 2: Atomic file write failure on Windows
After fixing issue 1, tests failed with:
Root Cause:
os.rename()fails on Windows when the destination file already exists. The code needs to useos.replace()which atomically replaces the destination on both Unix and Windows.Solution
Fix 1: Use
.as_posix()for virtual workspace mapping keysEnsure forward slashes on all platforms:
OperaPhenix (
openhcs/microscopes/opera_phenix.py):ImageXpress (
openhcs/microscopes/imagexpress.py):Fix 2: Use
os.replace()for atomic file writesatomic.py (
openhcs/io/atomic.py):Testing
This PR will trigger Windows CI to verify both fixes work on Windows. The mapping keys should now be stored with forward slashes in the JSON file, and atomic file writes should work correctly in multiprocessing mode.
Files Changed
openhcs/microscopes/opera_phenix.py- Use.as_posix()for mapping keysopenhcs/microscopes/imagexpress.py- Use.as_posix()for mapping keysopenhcs/io/atomic.py- Useos.replace()instead ofos.rename()