fix(desktop): normalize Linux Wayland/X11 backend and decoration policy

[bnema]

Closes #6258
Related to #6168

Summary

• Add a linux_windowing module (Rust) that handles backend selection and decoration policy
• Env overrides (OC_FORCE_X11, OC_FORCE_WAYLAND, OC_ALLOW_WAYLAND) take priority; then user settings; then auto-detection (Wayland-first, X11 fallback)
• Hide native window decorations when a tiling WM is detected (sway, hyprland, niri, i3, dwm, bspwm, river, qtile, xmonad, awesome, and others); keep decorations in full desktop environments (GNOME, KDE, XFCE, etc.)
• Users can override decoration behavior with OC_LINUX_DECORATIONS=native|none|auto
• Tiling WM detection works on both X11 and Wayland sessions

Why

On Niri (Wayland), OpenCode launched on XWayland by default unless the user forced Wayland explicitly. The new module makes backend selection explicit and predictable: Wayland first, X11 only as fallback. It also detects tiling window managers and hides decorations automatically, eliminating unwanted title bars in sessions like i3 on X11.

Verification

• Built the desktop app with bun run --cwd packages/desktop tauri build
• Launched the production binary: ./packages/desktop/src-tauri/target/release/OpenCode
• Verified on a Wayland tiling WM session:
  • Default startup falls back to XWayland correctly
  • OC_ALLOW_WAYLAND=1 starts native Wayland with no title bar decorations
• A SessionEnv struct makes the logic fully testable without touching real env vars
• Ran cargo test --manifest-path packages/desktop/src-tauri/Cargo.toml linux_windowing — 22 tests passed

[github-actions]

The following comment was made by an LLM, it may be inaccurate:

No duplicate PRs found

[github-actions]

Thanks for your contribution!

This PR doesn't have a linked issue. All PRs must reference an existing issue.

Please:

1. Open an issue describing the bug/feature (if one doesn't exist)
2. Add Fixes #<number> or Closes #<number> to this PR description

See CONTRIBUTING.md for details.

[Copilot]

Pull request overview

Aligns the Linux display-backend selection logic with an Electron/VS Code–style policy and introduces a Linux-specific window-decoration policy to avoid unwanted title bars (especially in tiling WMs).

Changes:

• Added a new Linux windowing policy module (linux_windowing) to centralize backend selection and decoration decisions (with unit tests).
• Updated Linux startup to apply the selected backend by setting the relevant environment variables and printing a plain note.
• Routed Tauri window creation through a decoration decision (cached) instead of always enabling decorations.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File	Description
packages/desktop/src-tauri/src/windows.rs	Applies a Linux-aware “use decorations” decision to window builders (cached via OnceLock).
packages/desktop/src-tauri/src/main.rs	Switches Linux backend selection to linux_windowing::select_backend and sets backend env vars accordingly.
packages/desktop/src-tauri/src/linux_windowing.rs	New module implementing backend selection + decoration policy (and tests).
packages/desktop/src-tauri/src/lib.rs	Exposes the new linux_windowing module on Linux builds.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Brendonovich]

Thanks for this, seems like a big improvement to our linux display handling. Works properly on my Ubuntu system.