feat: per-unit sync pipeline with incremental shortcut delivery

[christopherblaisdell]

Motivation

The current sync pipeline processes all platforms and collections as a single monolithic operation. Nothing appears in Steam until the entire sync finishes. For users with small libraries this is fine, but for users working with complete platform sets in RomM (hundreds or thousands of ROMs per platform across many platforms), the existing approach has serious problems:

1. Long wait with no feedback. A sync covering 5+ platforms with 300-700 ROMs each can take several minutes. The user sees a spinner the entire time with no indication of progress, no way to tell if it's working, and nothing usable until the very end.

2. All-or-nothing failure mode. If Steam's CEF browser crashes, the network drops, or the user cancels mid-sync, all work is lost - even platforms that were fully processed. The user has to start over from scratch.

3. CEF memory pressure. Emitting hundreds of shortcut-creation calls in a single burst can overwhelm Steam's embedded Chromium browser (steamwebhelper), causing crashes that wipe the entire shortcuts.vdf. We hit this in testing with just 362 Dreamcast ROMs at 20ms intervals.

4. No partial usability. Users managing large RomM libraries (retro gaming preservation, full MAME sets, complete console libraries) can't start playing games from Platform A while Platform B is still syncing.

The per-unit pipeline solves all of these by delivering value incrementally: each platform/collection is processed independently, shortcuts are registered in Steam as soon as that unit completes, and the backend moves to the next unit. If something fails on unit 3, units 1 and 2 are already in Steam and playable.

Architecture: Phase 0 as a minimally invasive approach

The key design decision was to introduce a Phase 0 (work queue construction) that runs before the existing sync logic, rather than rewriting the sync pipeline from scratch. Phase 0 calls _build_work_queue() to enumerate which platforms and collections need syncing - without fetching any ROM data - and produces a lightweight plan (unit name, type, ROM count estimate). The actual per-unit processing (ROM fetching, diffing, shortcut creation) reuses the same core logic that already existed; we're just scoping each invocation to a single platform or collection instead of processing everything at once.

This means:

• The existing sync steps are largely unchanged. ROM fetching, state diffing, shortcut creation, artwork downloading, and stale cleanup all work the same way they did before - they just operate on one unit at a time instead of the full library. This minimizes risk and keeps the change reviewable.
• The legacy preview/apply path is fully preserved. When Skip Preview is off, the original sync_preview -> sync_apply_delta flow runs exactly as before. The per-unit pipeline is an additive code path, not a replacement.
• Incremental skip is a natural consequence. Because each unit is evaluated independently, unchanged units can be detected and skipped without fetching their ROM lists at all - just compare the cached ROM count against RomM's platform metadata. This turns a 5-minute full rescan into a sub-second skip for platforms that haven't changed.

Performance implications

This architecture sets up a foundation for future performance work:

• Parallel unit processing. Today units are processed sequentially (safer for CEF), but the unit-based architecture makes it straightforward to process multiple units concurrently in a future PR - e.g., fetching ROM lists for platform B while the frontend is still creating shortcuts for platform A.
• Per-unit progress reporting. The frontend now receives granular sync_apply_unit events with unit index, total count, and ROM count. This enables real progress bars (unit 2/5, 64% of ROMs processed) instead of an indeterminate spinner.
• Targeted re-sync. Because the work queue is built dynamically, a future enhancement could allow users to re-sync a single platform without touching the rest - useful after adding new ROMs to one platform in RomM.
• Bounded memory usage. Processing one unit at a time means peak memory is proportional to the largest single platform, not the entire library. For users with 20+ platforms enabled, this is a significant reduction.

What changed

Backend (library.py)

• _build_work_queue(): Phase 0 builds the sync plan from enabled platforms/collections without fetching ROM lists - fast startup, no wasted API calls
• _sync_one_platform() / _sync_one_collection(): fetch ROMs, diff against state, emit sync_apply_unit event, wait for frontend callback before proceeding
• _wait_for_unit_results(): 60s heartbeat-based timeout with cancellation support - recovers gracefully if the frontend disconnects
• Incremental skip: unchanged units emit shortcuts from cached state without re-fetching from the RomM API

Frontend (syncManager.ts)

• processUnit(): handles sync_apply_unit events, creates/updates shortcuts with 50ms delay (CEF-safe), fetches artwork, reports results back to backend
• processStale(): handles stale shortcut cleanup after all units complete
• processLegacySyncApply(): preserves full backward compatibility with the existing preview/apply path

UI (MainPage.tsx)

• Skip Preview toggle now triggers the per-unit pipeline (startSync) directly
• Preview/Apply path remains unchanged when Skip Preview is off

Testing

Tested on Steam Deck with 3 units (2 platforms + 1 collection):

Unit	ROMs	Completed	Incremental?
Dreamcast	362	23:54:26	Yes - playable immediately
Nintendo 64	329	23:55:02	Yes - appeared while Metroid still syncing
Best of Metroid	11	23:55:14	Final unit

702 total shortcuts registered. Each unit appeared in Steam the moment it completed. No CEF crash at 50ms shortcut delay. 13 unit tests all passing.

New files

• docs/per-unit-pipeline.md - design document covering architecture, event flow, and open questions
• tests/services/test_per_unit_pipeline.py - 13 unit tests

Dependencies

This PR includes the changes from #215 (perf instrumentation). Recommend merging #215 first for a clean diff.