UI Anti-Duck-Typing Refactor: ABC-Based Architecture & Service Layer Extraction

OMERO_ZMQ_BACKEND_BUG.md

@@ -0,0 +1,221 @@
+# OMERO ZMQ Backend Selection Bug
+
+## Summary
+OMERO tests fail in ZMQ execution mode because checkpoint materialization attempts to create real directories using the disk backend instead of the OMERO backend, causing permission errors when trying to create directories under `/omero` (a virtual path).
+
+## Error Message
+```
+RuntimeError: ZMQ execution failed: Error creating directory /omero/plate_403_outputs/checkpoints_step0: [Errno 13] Permission denied: '/omero'
+```
+
+## Root Cause
+The `materialized_backend` in step plans is being set to `'disk'` instead of `'omero_local'` for OMERO plates when executing via ZMQ.
+
+## Architecture Context
+
+### OMERO Virtual Backend
+- OMERO uses **virtual paths** like `/omero/plate_123/` that don't exist on the filesystem
+- OMERO is a **virtual backend** - all paths are virtual, generated from OMERO plate structure
+- **No real filesystem operations** - `ensure_directory()` is a no-op for OMERO backend
+- OMERO output must be saved as **FileAnnotations** attached to OMERO objects, not as files
+- OMERO plates MUST use `omero_local` backend for both input and output
+
+### Backend Selection Flow
+1. **Test Configuration** (`tests/integration/test_main.py:343-344`):
+   ```python
+   if test_config.is_omero:
+       materialization_backend = MaterializationBackend('omero_local')
+   ```
+   Creates `GlobalPipelineConfig` with `VFSConfig(materialization_backend=MaterializationBackend.OMERO_LOCAL)`
+
+2. **Path Planning Phase** (`openhcs/core/pipeline/path_planner.py:185`):
+   ```python
+   'materialized_backend': self.vfs.materialization_backend.value,
+   ```
+   Sets initial `materialized_backend` to `self.vfs.materialization_backend.value`
+
+3. **Materialization Flag Planning Phase** (`openhcs/core/pipeline/materialization_flag_planner.py:83-85`):
+   ```python
+   if "materialized_output_dir" in step_plan:
+       materialization_backend = MaterializationFlagPlanner._resolve_materialization_backend(context, vfs_config)
+       step_plan["materialized_backend"] = materialization_backend
+   ```
+   Should overwrite `materialized_backend` with correct value from `vfs_config`
+
+4. **Execution** (`openhcs/core/steps/function_step.py:997`):
+   ```python
+   filemanager.ensure_directory(materialized_output_dir, materialized_backend)
+   ```
+   Uses `materialized_backend` from step plan to create checkpoint directories
+
+## ZMQ Configuration Transmission
+
+### Client Side (`openhcs/runtime/zmq_execution_client.py:82`):
+```python
+request['config_code'] = generate_config_code(global_config, GlobalPipelineConfig, clean_mode=True)
+```
+
+### Server Side (`openhcs/runtime/zmq_execution_server.py:196-198`):
+```python
+is_empty = 'GlobalPipelineConfig(\n\n)' in config_code or 'GlobalPipelineConfig()' in config_code
+global_config = GlobalPipelineConfig() if is_empty else (exec(config_code, ns := {}) or ns.get('config'))
+```
+
+### Config Serialization (`openhcs/debug/pickle_to_python.py:996-1028`):
+```python
+def generate_config_code(config, config_class, clean_mode=True):
+    """
+    Generate Python code representation of a config object.
+
+    Args:
+        config: Config instance (PipelineConfig, GlobalPipelineConfig, etc.)
+        config_class: The class of the config
+        clean_mode: If True, only show non-default values
+    """
+    config_repr = generate_clean_dataclass_repr(
+        config,
+        indent_level=0,
+        clean_mode=clean_mode,
+        required_imports=required_imports
+    )
+```
+
+## Investigation Results
+
+### Test 1: Config Serialization Works Correctly
+```python
+from openhcs.core.config import GlobalPipelineConfig, VFSConfig, MaterializationBackend
+from openhcs.debug.pickle_to_python import generate_config_code
+
+config = GlobalPipelineConfig(
+    vfs_config=VFSConfig(materialization_backend=MaterializationBackend.OMERO_LOCAL)
+)
+
+code = generate_config_code(config, GlobalPipelineConfig, clean_mode=True)
+print(code)
+```
+
+**Result**: Correctly generates:
+```python
+config = GlobalPipelineConfig(
+    vfs_config=VFSConfig(
+        materialization_backend=MaterializationBackend.OMERO_LOCAL
+    )
+)
+```
+
+### Test 2: Local OMERO Test Still Fails
+```bash
+pytest tests/integration/test_main.py --it-microscopes OMERO --it-zmq-mode zmq --it-visualizers none -v -s
+```
+
+**Result**: Both tests fail with same error:
+- `test_main[disk-OMERO-3d-multiprocessing-zmq-none]` - FAILED
+- `test_main[zarr-OMERO-3d-multiprocessing-zmq-none]` - FAILED
+
+Error: `Error creating directory /omero/plate_403_outputs/checkpoints_step0: [Errno 13] Permission denied: '/omero'`
+
+## Attempted Fix (Did Not Work)
+
+### Change Made
+Added `vfs_config=None` to `PipelineConfig` in ZMQ mode to inherit from global config:
+
+```python
+# tests/integration/test_main.py:591-597
+pipeline_config = PipelineConfig(
+    path_planning_config=LazyPathPlanningConfig(
+        output_dir_suffix=CONSTANTS.OUTPUT_SUFFIX
+    ),
+    step_well_filter_config=LazyStepWellFilterConfig(well_filter=CONSTANTS.PIPELINE_STEP_WELL_FILTER_TEST),
+    vfs_config=None,  # Inherit from global config
+)
+```
+
+**Commit**: `8f01ee5` - "fix: Set vfs_config=None in ZMQ pipeline config to inherit OMERO backend from global config"
+
+**Result**: Tests still fail with same error. This suggests the issue is NOT in the test configuration.
+
+## Hypotheses to Investigate
+
+### Hypothesis 1: Config Inheritance Not Working in ZMQ Server
+The ZMQ server might be creating a new orchestrator that doesn't properly inherit the global config's `vfs_config`. Check:
+- How does the server create the orchestrator from the deserialized config?
+- Is the `vfs_config` from `GlobalPipelineConfig` being passed to the orchestrator's `pipeline_config`?
+- Does the config context manager properly propagate `vfs_config` from global to pipeline config?
+
+### Hypothesis 2: Clean Mode Dropping None Values
+When `pipeline_config` has `vfs_config=None` and is serialized with `clean_mode=True`, the `None` value might be dropped, causing the server to use the default `VFSConfig()` with `materialization_backend=DISK`. Check:
+- Does `generate_clean_dataclass_repr()` preserve `None` values in clean mode?
+- What does the actual serialized `pipeline_config_code` look like?
+- Add debug logging to print the generated config code before sending to server
+
+### Hypothesis 3: Lazy Config Resolution Issue
+The lazy config system might not be properly resolving `vfs_config=None` to inherit from the global config. Check:
+- Does `LazyVFSConfig` exist and is it being used?
+- How does the config context manager handle `None` values in lazy configs?
+- Is there a difference between `vfs_config=None` and `vfs_config=LazyVFSConfig()`?
+
+### Hypothesis 4: Server-Side Config Merging
+The server might be merging the global config and pipeline config incorrectly. Check:
+- How does `_execute_with_orchestrator()` merge the configs?
+- Is the `vfs_config` from `pipeline_config` overriding the one from `global_config`?
+- Does the orchestrator use the correct config hierarchy?
+
+## Debugging Steps
+
+1. **Add debug logging to print generated config code**:
+   ```python
+   # In zmq_execution_client.py:82
+   config_code = generate_config_code(global_config, GlobalPipelineConfig, clean_mode=True)
+   logger.info(f"Generated global config code:\n{config_code}")
+
+   pipeline_config_code = generate_config_code(pipeline_config, PipelineConfig, clean_mode=True)
+   logger.info(f"Generated pipeline config code:\n{pipeline_config_code}")
+   ```
+
+2. **Add debug logging on server side to print received config**:
+   ```python
+   # In zmq_execution_server.py:196
+   logger.info(f"Received config code:\n{config_code}")
+   logger.info(f"Received pipeline config code:\n{pipeline_config_code}")
+   ```
+
+3. **Add debug logging in MaterializationFlagPlanner**:
+   ```python
+   # In materialization_flag_planner.py:84
+   logger.info(f"Step {i}: vfs_config.materialization_backend = {vfs_config.materialization_backend}")
+   logger.info(f"Step {i}: resolved materialization_backend = {materialization_backend}")
+   ```
+
+4. **Check what vfs_config the orchestrator is using**:
+   ```python
+   # In orchestrator initialization
+   logger.info(f"Orchestrator vfs_config: {self.pipeline_config.vfs_config}")
+   logger.info(f"Orchestrator materialization_backend: {self.pipeline_config.vfs_config.materialization_backend}")
+   ```
+
+## Files Involved
+
+- `tests/integration/test_main.py` - Test configuration
+- `openhcs/core/config.py` - Config dataclasses and defaults
+- `openhcs/core/pipeline/path_planner.py` - Sets initial `materialized_backend`
+- `openhcs/core/pipeline/materialization_flag_planner.py` - Resolves final `materialized_backend`
+- `openhcs/core/steps/function_step.py` - Uses `materialized_backend` for checkpoint creation
+- `openhcs/runtime/zmq_execution_client.py` - Serializes and sends config
+- `openhcs/runtime/zmq_execution_server.py` - Deserializes and uses config
+- `openhcs/debug/pickle_to_python.py` - Config serialization logic
+- `openhcs/io/disk.py` - Disk backend `ensure_directory()` (raises the error)
+- `openhcs/io/omero_local.py` - OMERO backend `ensure_directory()` (no-op)
+
+## Expected Behavior
+For OMERO tests in ZMQ mode:
+1. Test creates `GlobalPipelineConfig` with `VFSConfig(materialization_backend=MaterializationBackend.OMERO_LOCAL)`
+2. Config is serialized and sent to ZMQ server
+3. Server deserializes config and creates orchestrator
+4. Orchestrator compiles pipeline with correct `materialized_backend='omero_local'` in step plans
+5. Checkpoint materialization calls `filemanager.ensure_directory(dir, 'omero_local')`
+6. OMERO backend's no-op `ensure_directory()` is called (no error)
+
+## Actual Behavior
+Steps 1-3 appear to work correctly, but step 4 sets `materialized_backend='disk'` instead of `'omero_local'`, causing step 5 to call the disk backend's `ensure_directory()` which tries to create a real directory at `/omero/...` and fails with permission denied.
+

THREAD_LOCAL_GLOBAL_ARCHITECTURE.md

@@ -0,0 +1,162 @@
+# Thread-Local Global Config Architecture Analysis
+
+## Current Architecture
+
+### What is `thread_local_global`?
+
+`thread_local_global` (returned by `get_base_global_config()`) is a **thread-local singleton** that stores the "base" GlobalPipelineConfig instance. It's stored in `_global_config_contexts` dict in `global_config.py`.
+
+### When is it set?
+
+1. **App startup** (`app.py` line 95):
+   ```python
+   set_global_config_for_editing(GlobalPipelineConfig, self.global_config)
+   ```
+   - Happens once when the GUI app initializes
+   - Sets the cached/default GlobalPipelineConfig
+
+2. **When user saves global config** (`main.py` line 584):
+   ```python
+   set_global_config_for_editing(GlobalPipelineConfig, new_config)
+   ```
+   - Updates the thread-local when user saves changes to GlobalPipelineConfig
+
+### When is it used?
+
+**GLOBAL_LIVE_VALUES layer** (`context_layer_builders.py` line 356-364):
+```python
+thread_local_global = get_base_global_config()
+global_live_values = _reconstruct_nested_dataclasses(global_live_values, thread_local_global)
+global_live_instance = dataclasses.replace(thread_local_global, **global_live_values)
+```
+
+**Purpose**: Merge live values from the GlobalPipelineConfig manager into the thread-local base config.
+
+### The Problem
+
+**`thread_local_global` is NEVER updated when nested managers change!**
+
+Flow when you edit `well_filter_config.well_filter`:
+1. User types "3" → `well_filter_config` manager updates
+2. `thread_local_global` still has `WellFilterConfig(well_filter=None)` (default)
+3. User backspaces → `well_filter_config.well_filter` becomes None
+4. When building GLOBAL_LIVE_VALUES for siblings:
+   - Gets `{'well_filter_config': (WellFilterConfig, {'well_filter': None})}`
+   - Filters out None → empty dict
+   - Doesn't update `well_filter_config` field
+   - `thread_local_global` still has old `WellFilterConfig(well_filter=None)`
+   - But wait - it should have `well_filter=3` from step 1!
+
+**Actually, the issue is different**: `thread_local_global` is only updated when you **save** the GlobalPipelineConfig. So it has the LAST SAVED state, not the current editing state.
+
+When you type "3" in `well_filter_config.well_filter`:
+- The GlobalPipelineConfig manager's widgets show "3"
+- But `thread_local_global` still has the old saved value (or default)
+- When you backspace and we filter out None, we keep the stale value from `thread_local_global`
+
+## Three Proposed Solutions
+
+### Option 1: Update `thread_local_global` when nested managers change
+
+**Approach**: Call `set_global_config_for_editing()` whenever a nested manager in GlobalPipelineConfig changes.
+
+**Implementation**:
+```python
+# In parameter_form_manager.py, _on_nested_parameter_changed()
+if self.config.is_global_config_editing and self.global_config_type is not None:
+    # Rebuild GlobalPipelineConfig from current values
+    current_values = self.get_current_values()
+    new_global_config = self.global_config_type(**current_values)
+    set_global_config_for_editing(self.global_config_type, new_global_config)
+```
+
+**Pros**:
+- ✅ Keeps `thread_local_global` in sync with current editing state
+- ✅ Minimal changes to context layer building logic
+- ✅ Maintains existing architecture
+
+**Cons**:
+- ❌ **Violates the purpose of `thread_local_global`** - it's supposed to be the "saved" config, not the "currently editing" config
+- ❌ **Performance**: Rebuilding GlobalPipelineConfig on every nested change is expensive
+- ❌ **Semantic confusion**: `set_global_config_for_editing()` is documented as "ONLY for legitimate global config modifications" (saving, loading), not for every keystroke
+- ❌ **Side effects**: Other parts of the system might rely on `thread_local_global` being the saved state
+
+### Option 2: Don't use `thread_local_global` as base for GLOBAL_LIVE_VALUES
+
+**Approach**: Build GLOBAL_LIVE_VALUES layer entirely from the GlobalPipelineConfig manager's current values, without merging into `thread_local_global`.
+
+**Implementation**:
+```python
+# In context_layer_builders.py, GlobalLiveValuesBuilder.build()
+global_live_values = _get_manager_values(global_manager, use_user_modified_only)
+
+# Build fresh GlobalPipelineConfig from current manager values
+# Don't use thread_local_global at all
+global_live_instance = manager.global_config_type(**global_live_values)
+
+return ContextLayer(layer_type=self._layer_type, instance=global_live_instance)
+```
+
+**Pros**:
+- ✅ **Correct semantics**: GLOBAL_LIVE_VALUES truly reflects the current live state
+- ✅ **No stale data**: Always uses current manager values
+- ✅ **Simpler logic**: No need for `_reconstruct_nested_dataclasses()` merging
+- ✅ **Preserves `thread_local_global` purpose**: It remains the "saved" config
+
+**Cons**:
+- ❌ **Incomplete data**: `get_current_values()` might not include all fields (only user-modified or visible fields)
+- ❌ **Dataclass construction**: Can't construct GlobalPipelineConfig if required fields are missing
+- ❌ **Loss of non-user-set fields**: Fields that weren't edited won't be in the layer
+
+### Option 3: Track user-cleared fields separately and explicitly clear them
+
+**Approach**: Add a new tracking set `_user_cleared_fields` to distinguish between "never set (None)" and "user cleared to None".
+
+**Implementation**:
+```python
+# In parameter_form_manager.py
+self._user_cleared_fields = set()  # New tracking
+
+# When user backspaces to clear:
+def _on_widget_value_changed(self, param_name, value):
+    if value is None and param_name in self._user_set_fields:
+        self._user_cleared_fields.add(param_name)
+    # ...
+
+# In _reconstruct_nested_dataclasses():
+if field_name in user_cleared_fields:
+    # User explicitly cleared this - mark for removal
+    reconstructed[field_name] = CLEARED_SENTINEL
+```
+
+Then in context merging, handle `CLEARED_SENTINEL` specially to remove stale values.
+
+**Pros**:
+- ✅ **Precise semantics**: Distinguishes "never set" from "user cleared"
+- ✅ **Correct behavior**: User-cleared fields properly override stale values
+- ✅ **Preserves inheritance**: Non-cleared None values still inherit
+
+**Cons**:
+- ❌ **Complex**: Requires new tracking mechanism and sentinel values
+- ❌ **Invasive**: Changes needed in multiple places (tracking, reconstruction, merging)
+- ❌ **Maintenance burden**: More state to track and keep consistent
+- ❌ **Edge cases**: What happens when user clears, then sets again, then clears?
+
+## Recommendation
+
+**Option 2** is the cleanest solution because:
+
+1. **Semantic correctness**: GLOBAL_LIVE_VALUES should reflect the current live editing state, not the saved state
+2. **Architectural clarity**: Separates "saved config" (`thread_local_global`) from "currently editing config" (manager values)
+3. **Simplicity**: Removes the need for complex merging logic
+
+However, we need to handle the "incomplete data" problem. The solution is to use `get_current_values()` (not `get_user_modified_values()`) which includes ALL fields, and reconstruct nested dataclasses from the nested managers' current values.
+
+## Implementation Plan for Option 2
+
+1. Modify `GlobalLiveValuesBuilder.build()` to build fresh GlobalPipelineConfig from manager's current values
+2. Use `get_current_values()` to get ALL fields (not just user-modified)
+3. Reconstruct nested dataclasses from nested managers' current values
+4. Don't merge into `thread_local_global` at all
+5. Keep filtering out None values in nested dataclasses to preserve inheritance
+