WinUI tray menu crashes in Microsoft.UI.Windowing.Core.dll after idle/interaction

[shanselman]

Problem

The WinUI 3 tray app crashes intermittently when clicking the tray icon to show the menu. The crash occurs in the native WinUI windowing layer and bypasses all .NET exception handlers.

Crash Details

• Faulting module: Microsoft.UI.Windowing.Core.dll (version 10.0.27108.1025)
• Exception code: 0xe0464645 (CLR exception marker)
• Fault offset: 0x000000000000a280 (consistent across all crashes)
• Platform: Windows 11 ARM64
• Windows App SDK: 1.8.250906003
• WinUIEx: 2.9.0
• .NET: 9.0 (targeting windows10.0.19041.0)

Reproduction Steps

1. Start the tray app
2. Click tray icon - menu appears (works)
3. Click away to dismiss menu
4. Wait a few seconds OR open Settings window and close it
5. Click tray icon again - CRASH

The crash doesn't happen immediately but becomes more likely after:

• The app has been idle for a period
• Other windows have been opened/closed (like Settings)
• Multiple menu show/hide cycles

What We've Tried

1. Window Reuse Pattern

Instead of creating/destroying TrayMenuWindow each click, tried reusing a single window:

• Hide() instead of Close() on deactivation
• ClearItems() and rebuild menu content
• Result: Black square appeared instead of menu content

2. UI Thread Dispatching

Added checks to ensure tray icon callbacks dispatch to UI thread

• Result: Still crashes

3. Crash Logging

Added UnhandledException, DomainUnhandledException, UnobservedTaskException handlers

• Result: No crash log written - crash is in native code

4. Backdrop Removal

Removed DesktopAcrylicBackdrop from TrayMenuWindow

• Result: Still crashes

5. Async Data Fetch with Timeout

Added brief wait for gateway data before showing menu

• Result: Still crashes

Current Architecture

TrayMenuWindow extends WinUIEx.WindowEx, configured as popup-style (no title bar, always on top, closes on deactivation). ShowAtCursor() uses P/Invoke to position near taskbar.

App creates new TrayMenuWindow on each tray click, closes previous one first.

Hypothesis

The crash appears to be related to WinUI window lifecycle management:

1. GC collecting something the native windowing layer still references after idle
2. COM apartment threading issue when tray icon callback fires
3. WindowEx/WinUIEx issue with rapid window creation/destruction
4. Windows App SDK bug with unpackaged apps and system tray interaction

Potential Solutions to Explore

1. MenuFlyout approach: Use e.Flyout = BuildMenuFlyout() in tray callbacks instead of custom window
2. Keep-alive window: Create an invisible window to keep WinUI runtime active
3. GC.KeepAlive: Pin critical objects to prevent collection

Files of Interest

• src/OpenClaw.Tray.WinUI/App.xaml.cs - Main app, tray icon handling
• src/OpenClaw.Tray.WinUI/Windows/TrayMenuWindow.xaml.cs - Popup menu window

Windows Event Log Entry

Faulting application name: OpenClaw.Tray.WinUI.exe
Faulting module name: Microsoft.UI.Windowing.Core.dll
Exception code: 0xe0464645
Fault offset: 0x000000000000a280

[shanselman]

Deep Analysis for Cloud Agent

The Core Problem

WinUI 3 was designed assuming a main window exists. This tray-only app (no main window) is an edge case where:

• The WinUI runtime may not stay fully initialized
• Thread affinity may be lost after idle
• GC may collect objects the native layer still references

Exception Code Analysis

0xe0464645 - The bytes 46 46 45 spell "FFE" which is the CLR exception marker. This means a .NET exception propagated into native code and wasn't caught. The crash is happening during native window initialization, after our managed code calls new TrayMenuWindow() but before control returns.

Why Window Reuse Showed Black Square

When we tried Hide() instead of Close(), the window reappeared as black. This suggests:

1. The XAML visual tree gets detached when hidden
2. The DirectComposition swap chain becomes invalid
3. The content isn't being re-rendered on show

This is actually a clue - the window DOES exist, but its rendering pipeline is broken.

WinUIEx TrayIcon Architecture

WinUIEx's TrayIcon creates an internal hidden message-only window to receive Shell_NotifyIcon callbacks. When user clicks, this fires on whatever thread the message pump runs on. The question: is this thread the same STA thread where we created the DispatcherQueue?

Most Promising Solutions (Priority Order)

1. MenuFlyout Approach (Partially Implemented)

Use e.Flyout = BuildMenuFlyout() in tray callbacks. This attaches a flyout to WinUIEx's internal window instead of creating our own. No window creation = no crash.

Status: I started this in App.xaml.cs (BuildTrayMenuFlyout method). Needs testing.

2. Keep-Alive Hidden Window

Create a hidden window at startup that stays alive forever:

// In OnLaunched, after InitializeComponent
_keepAliveWindow = new Window();
_keepAliveWindow.Content = new Grid();
_keepAliveWindow.AppWindow.IsShownInSwitchers = false;
// Position off-screen and minimize
_keepAliveWindow.AppWindow.Move(new Windows.Graphics.PointInt32(-10000, -10000));

This keeps the WinUI runtime properly initialized.

3. Pin DispatcherQueue Reference

The crash might be GC collecting the DispatcherQueue after idle:

// Make it a static field that GC won't touch
private static DispatcherQueue? _pinnedDispatcherQueue;

// In OnLaunched
_pinnedDispatcherQueue = DispatcherQueue.GetForCurrentThread();
_dispatcherQueue = _pinnedDispatcherQueue;

4. Investigate WinUIEx Threading

Look at WinUIEx source: https://github.com/dotMorten/WinUIEx
Check TrayIcon.cs for how it handles the Shell_NotifyIcon callback threading.

Key Files to Modify

• src/OpenClaw.Tray.WinUI/App.xaml.cs lines 178-290 (tray handling)
• src/OpenClaw.Tray.WinUI/Windows/TrayMenuWindow.xaml.cs (popup window)

Testing Approach

1. Start app
2. Click tray icon 2-3 times quickly
3. Open Settings, close it
4. Wait 30 seconds
5. Click tray icon
6. If no crash after 10 cycles, likely fixed

[shanselman]

Fixed in v0.3.0. The crash was caused by WinUI 3's MenuFlyout requiring a valid UIElement context when showing from a tray icon callback. The fix uses a TrayMenuAnchorWindow (invisible 1x1 window) to anchor the MenuFlyout, preventing the native crash in Microsoft.UI.Windowing.Core.dll. See \src/OpenClaw.Tray.WinUI/Windows/TrayMenuAnchorWindow.xaml.cs\ and the updated tray menu code in \App.xaml.cs. Has been stable through extensive testing across multiple sessions.