Skip to content

Fix Windows virtual workspace path separator issue#33

Merged
trissim merged 14 commits intomainfrom
fix/windows-virtual-workspace-path-separators
Oct 24, 2025
Merged

Fix Windows virtual workspace path separator issue#33
trissim merged 14 commits intomainfrom
fix/windows-virtual-workspace-path-separators

Conversation

@trissim
Copy link
Collaborator

@trissim trissim commented Oct 24, 2025

Problem

Windows CI tests were failing with two separate issues:

Issue 1: Virtual workspace path separator mismatch

The virtual workspace mapping keys were being created with backslashes on Windows, causing "No wells found for processing" errors.

Root Cause: When building the workspace_mapping dictionary, the code used str(Path(...)) which converts to backslashes on Windows:

  • Windows: 'Images\\r01c01f007p001-ch1sk1fk1fl1.tiff'
  • Linux: 'Images/r01c01f007p001-ch1sk1fk1fl1.tiff'

The mapping lookup code normalizes incoming paths to forward slashes with .replace('\\', '/'), but the mapping keys themselves had backslashes on Windows, causing all lookups to fail.

Issue 2: Atomic file write failure on Windows

After fixing issue 1, tests failed with:

[WinError 183] Cannot create a file when that file already exists:
'.tmpopenhcs_metadata.json0g40hqyz.json' -> 'openhcs_metadata.json'

Root Cause: os.rename() fails on Windows when the destination file already exists. The code needs to use os.replace() which atomically replaces the destination on both Unix and Windows.

Solution

Fix 1: Use .as_posix() for virtual workspace mapping keys

Ensure forward slashes on all platforms:

OperaPhenix (openhcs/microscopes/opera_phenix.py):

# Before
virtual_relative = str(Path("Images") / new_name)
real_relative = str(Path("Images") / file_name)

# After
virtual_relative = (Path("Images") / new_name).as_posix()
real_relative = (Path("Images") / file_name).as_posix()

ImageXpress (openhcs/microscopes/imagexpress.py):

# Before
virtual_relative = str(directory.relative_to(plate_path) / new_filename)
real_relative = str(Path(img_file).relative_to(plate_path))

# After
virtual_relative = (directory.relative_to(plate_path) / new_filename).as_posix()
real_relative = Path(img_file).relative_to(plate_path).as_posix()

Fix 2: Use os.replace() for atomic file writes

atomic.py (openhcs/io/atomic.py):

# Before
os.rename(tmp_path, str(file_path))

# After
os.replace(tmp_path, str(file_path))  # Atomic on both Unix and Windows

Testing

This PR will trigger Windows CI to verify both fixes work on Windows. The mapping keys should now be stored with forward slashes in the JSON file, and atomic file writes should work correctly in multiprocessing mode.

Files Changed

  • openhcs/microscopes/opera_phenix.py - Use .as_posix() for mapping keys
  • openhcs/microscopes/imagexpress.py - Use .as_posix() for mapping keys
  • openhcs/io/atomic.py - Use os.replace() instead of os.rename()

…ward slashes on Windows

The virtual workspace mapping keys were being created with str(Path(...)) which
uses backslashes on Windows. This caused all mapping lookups to fail on Windows
because the JSON stored keys with backslashes while the lookup code normalized
incoming paths to forward slashes.

Changes:
- OperaPhenix: Use .as_posix() when building virtual_relative and real_relative paths
- ImageXpress: Use .as_posix() when building virtual_relative and real_relative paths

This ensures the mapping keys are always stored with forward slashes in the JSON
file regardless of platform, matching the normalization done during lookups.

Fixes Windows CI failures where orchestrator found 0 wells due to failed file listings.
…n Windows

os.rename() fails on Windows when the destination file already exists with
WinError 183 'Cannot create a file when that file already exists'.

os.replace() is the atomic replacement operation that works on both Unix and
Windows platforms - it atomically replaces the destination file if it exists.

This fixes the atomic metadata write failures in multiprocessing mode on Windows
where multiple processes were trying to update the same metadata file.

Changes:
- atomic.py: Replace os.rename() with os.replace() in atomic_write_json()

Fixes Windows CI error:
[WinError 183] Cannot create a file when that file already exists:
'.tmpopenhcs_metadata.json0g40hqyz.json' -> 'openhcs_metadata.json'
…pattern_path

The build_dict_pattern_path() function was hardcoding '/' for path splitting,
which failed on Windows where paths use backslashes.

Changed from:
  dir_part, filename = base_path.rsplit('/', 1)
  return f"{dir_part}/{well_id}_w{dict_key}_{rest}"

To:
  path = Path(base_path)
  dir_part = path.parent
  filename = path.name
  return str(dir_part / f"{well_id}_w{dict_key}_{rest}")

This fixes the error:
ValueError: not enough values to unpack (expected 2, got 1)

Fixes Windows CI error in _filter_special_outputs_for_function when building
channel-specific paths for dict pattern special outputs.
Add comprehensive GUI testing for PyQt6 interface across Linux, Windows, and macOS.

Features:
- Tests on Python 3.11 and 3.12
- Linux: Uses Xvfb for headless Qt testing
- Windows: Uses offscreen platform for headless testing
- macOS: Uses offscreen platform for headless testing
- Uploads test artifacts (logs, screenshots) on failure
- Runs on push to main, PRs, and manual dispatch

Test configuration:
- QT_QPA_PLATFORM=offscreen for headless mode
- OPENHCS_CPU_ONLY=true to skip GPU-dependent tests
- --maxfail=5 to stop after 5 failures for faster feedback

This ensures GUI code works correctly across all platforms and catches
cross-platform issues like path separator problems early.
…table names

OMERO table names cannot contain dots except for the .h5 extension.
Previously, output_path.stem only removed the last extension (.json),
leaving intermediate extensions like .roi.zip in the table name.

For example:
- Path: A01_s001_w1_z001_t001_cell_counts_step7.roi.zip.json
- Old: table_name = 'A01_s001_w1_z001_t001_cell_counts_step7.roi.zip'
- New: table_name = 'A01_s001_w1_z001_t001_cell_counts_step7'

This caused OMERO to reject the table creation with:
  omero.InternalException: message = null table as argument

Fix: Use output_path.name.split('.')[0] to remove ALL extensions.
Expand test coverage to include Python 3.13 and 3.14 for both Linux and Windows CI.

Changes:
- integration-tests-focused: Test on Python 3.11, 3.12, 3.13, 3.14
- integration-tests-windows: Test on Python 3.11, 3.12, 3.13, 3.14

This ensures OpenHCS works correctly across all supported Python versions
and catches version-specific issues early.
Pin NumPy < 2.2 for Python 3.14 to work around histogram/bincount bug.
NumPy 2.2.x has a compatibility issue with Python 3.14 that causes
ValueError in scikit-image's threshold_otsu function.

Re-enabled Python 3.14 in CI matrix for both Linux and Windows tests.
Reduce CI jobs from 32 to 10 using smart grouping:

Group 1: Python boundary tests (4 jobs)
- Test Python 3.11 and 3.14 on both Linux and Windows
- Validates oldest and newest supported versions
- Catches dependency compatibility issues (like NumPy 3.14 bug)

Group 2: Backend/microscope coverage (4 jobs)
- Test all backend/microscope combos on both OSes with Python 3.12
- Ensures cross-platform compatibility for all backends

Group 3: OMERO tests (2 jobs)
- Test OMERO on Linux with Python 3.11 and 3.14
- Linux only (OMERO requires Docker)

Total: 10 jobs (down from 32)
Coverage: All Python versions, both OSes, all backends, all microscopes

Created issue #34 to track NumPy version pin removal.
zeroc-ice fails to build from source on Python 3.11+ due to setuptools
compatibility issues. Use pre-built wheels from Glencoe Software (OMERO
maintainers) instead:

- Python 3.11: zeroc-ice 3.6.5 (cp311, manylinux_2_28)
- Python 3.14: zeroc-ice 3.6.5 (cp314, manylinux_2_28)

This matches the approach used by napari and other OMERO-dependent projects.
Python 3.14 is too new - zeroc-ice pre-built wheels only available up to
Python 3.12 (as of 20240202 release). Testing on 3.11 and 3.13 provides
good coverage while we wait for OMERO ecosystem to catch up.

Total CI jobs: 11 (4 boundary + 4 backend/microscope + 2 OMERO + 1 wheel test)
zeroc-ice pre-built wheels only available up to Python 3.12 (as of
20240202 release). Python 3.13 and 3.14 not yet supported by OMERO
ecosystem.

Testing on 3.11 (oldest) and 3.12 (newest supported) provides boundary
coverage for OMERO integration.

Total CI jobs: 11
Python 3.14 is too new - no pre-built wheels available:
- NumPy: builds from source, stalls on Windows CI (5+ min timeout)
- zeroc-ice (OMERO): no wheels at all

Changed boundary testing from 3.11/3.14 to 3.11/3.13 (well-supported versions).

Total CI jobs: 11 (4 boundary + 4 backend/microscope + 2 OMERO + 1 wheel)

Python 3.14 support will be added when ecosystem catches up (likely Q1 2025).
Added comprehensive header comment documenting:
- 11 total jobs organized into 4 groups
- Boundary testing strategy (3.11, 3.13)
- Python 3.14 exclusion rationale
- OMERO limitations (Python 3.11-3.12 only)
- Coverage: all Python versions, OSes, backends, microscopes
@trissim trissim merged commit 4431356 into main Oct 24, 2025
19 checks passed
@trissim trissim deleted the fix/windows-virtual-workspace-path-separators branch October 30, 2025 17:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant