MacOS can't wake up during resize, breaks timeout schemes

[willglynn]

As the comment describes, nextEventMatchingMask:untilDate:inMode:dequeue: blocks during a resize and does not return events until the resize is complete. That includes not returning events sent via postEvent:atStart:, which means Proxy::wakeup() doesn't actually cause run_forever() to wake up.

This property is especially unfortunate with glutin_window, which implements its own fn wait_event_timeout(&mut self, timeout: Duration) by calling run_forever() and interrupting when the timeout expires. Since it can't interrupt, a wait_event_timeout() that wants to yield 60 fps to e.g. drive a render loop will simply hang entirely until the user is done resizing.

I've been experimenting with ways to get nextEventMatchingMask: to be less thorny, but another complimentary approach would be to expose a new run_until_timeout() call in addition to poll_events() and run_forever() -- or perhaps to deprecate poll_events() and run_forever() in favor of run_until_timeout(Some(0)) and run_until_timeout(None). A timeout-aware API would let glutin_window pass its deadline all the way down to nextEventMatchingMask:untilDate:, avoid spawning needless timeout threads, and sidestep some of the resize-related blocking issues all at once.

[mitchmindtree]

As far as I understand, this modal resize event loop is an unfortunate wart stemming from macos history 😿 I've spent quite a bit of time looking into solutions but haven't personally had much luck - hopefully your endeavour turns out better than mine!

One of the particularly frustrating things about this is that, to properly handle live-resize a winit application's drawing must be abstracted in a way that allows it to be called when Resized events are received on top of the regular place at which drawing is done in the application. This becomes particularly tedious when the drawing needs to be timed (e.g. don't draw more than 60fps) which is in most cases as it is common for large numbers of Resized events to be emitted at once.

All that said, the current situation is much better than pre #126 days, where it was impossible to handle live resize without registering a totally separate callback function that could only share data with the main application via global state 😱 Hopefully we can work out some way to improve things even further though.

I'm not sure about adding a timeout method, or whether it will even help much with our macos problem. In theory, this should already be easy for users to do by using a timer to call EventsLoopProxy::wakeup, interrupting the run_forever method - unfortunately it's only macos that has problems with this due to this problem of blocking on resize that you've already outlined. I'm skeptical that specifying a timeout for nextEventMatchingMask will actually help in this case, as from what I remember the current implementation of poll_events is basically this with a timeout of 0 and it still blocks during resize.

[willglynn]

All that said, the current situation is much better than pre #126 days

That's why I'm looking at this now and not earlier :-)

I'm not sure about adding a timeout method, or whether it will even help much with our macos problem.

A timeout-capable API would remove this pattern:

let events_loop_proxy = self.events_loop.create_proxy();
thread::spawn(move || {
    thread::sleep(timeout);
    events_loop_proxy.wakeup().ok();
});
{
    let ref mut events = self.events;
    self.events_loop.run_forever(|ev| {
        events.push_back(ev);
        glutin::ControlFlow::Break
    });
}

That's pistoncore-glutin_window implementing a timeout-oriented API on top of winit's timeout-less API which is implemented on top of e.g. MacOS's timeout-oriented API.

Letting the caller express timeouts seems intrinsically useful: if it's a timed event loop, the caller wants to either a) receive a message from the windowing system or b) get control again without a message so it can start another draw cycle. Every platform should have a way of accommodating this, and it's likely better than unconditionally spawning a thread for every message received. If not, maybe the platform should be responsible for implementing timeouts by spawning a thread and waking itself.

"Receive with timeout" is also a more fundamental approach than the current APIs: poll_events() is a special case where you want to receive any messages that are available right now without blocking (zero timeout), and run_forever() is a special case where you want to keep running until you want to receive a message (infinite timeout).

I'm skeptical that specifying a timeout for nextEventMatchingMask will actually help in this case, as from what I remember the current implementation of poll_events is basically this with a timeout of 0 and it still blocks during resize.

Yeah… the situation is worse than the comment in the code, and a timeout here doesn't help as much as I initially thought.

Resizing does block on _nextEventMatchingEventMask:untilDate:inMode:dequeue:, but not where poll_events()/run_forever() tries to receive a message. After receiving the message that starts a resize (I think it's a general mouse down event?), both poll_event() and run_forever():

• call ns_event_to_event(),
• which calls -[NSApplication(NSEvent) sendEvent:],
• which calls -[NSWindow(NSEventRouting) sendEvent:],
• which calls -[NSWindow(NSEventRouting) _reallySendEvent:isDelayedEvent:],
• which calls -[NSWindow(NSEventRouting) _handleMouseDownEvent:isDelayedEvent:],
• which calls -[NSThemeFrame mouseDown:],
• which calls -[NSThemeFrame handleMouseDown:],
• which calls -[NSTitledFrame attemptResizeWithEvent:],
• which calls -[NSWindow(NSWindowResizing) _resizeWithEvent:],
• which calls -[NSView _getNextResizeEventFromMask:invalidatingLiveResizeCacheIfNecessary:],
• which calls -[NSApplication(NSEvent) _nextEventMatchingEventMask:untilDate:inMode:dequeue:]

That's bad. For resize events, sendEvent: wanders into an internal runloop that doesn't return until the resize is complete. This is where we get stuck.

A timeout on the outer loop won't fix that, but let's say the caller told us when they needed control again.

How would that help? Well, I don't think we can avoid this inner runloop, but a runloop observer could punt control back to user code before that inner runloop goes to sleep or after it wakes. We should be able to wake up the event loop with a timer, as @mitchmindtree point outs, even if the timer message gets ignored by the mask and therefore not delivered in a timely fashion.

How would that help? Well… let's say we somehow manage to get back to winit from deep inside this context, and we see that the user's timeout has expired. That tells us that we need to return from the poll_events() or run_until_timeout() call immediately. We can't unwind the stack back to winit's caller, so we need to avoid piling on top of it in the first place. Spawn a thread, right? Well, we also need to be on the caller's thread. That leaves stack switching: call sendEvent: in a coroutine which either a) runs to completion or b) yields when a timeout expires.

This would let winit's event loop return to the user on time even though sendEvent: hasn't yet returned. The user gets control, draws something, calls the event loop again. Event loop sees there's still a sendEvent: call pending, and instead of working the normal runloop again, it resumes the coroutine which stack switches back to the runloop observer callback in the middle of -[NSTitledFrame attemptResizeWithEvent:]'s filtered runloop. Maybe sendEvent: returns and life goes back to normal, or maybe it doesn't and we suspend the coroutine again and repeat.

[willglynn]

I have a coroutines branch that accomplishes most of what I described.

First, I started with refactoring: the MacOS event loop now has a CocoaEvent enum that tracks Received from Cocoa -> Sending back to Cocoa -> Complete (and possibly translated into an events::Event) lifecycle. I also refactored the "send event" activity into a struct that possibly requires work() to be called several times, and made the general event loop aware of this. The basic implementation calls sendEvent: as usual.

Second, I added an optional Mac dependency on context which can provide suitable coroutine glue. As of this point, winit uses the basic mod send_event implementation by default, but --feature context will pull in context and use mod send_event_context instead. I then got it calling sendEvent: from inside the coroutine, but still blocking as usual.

Third, I added an observer to the default runloop triggering on kCFRunLoopBeforeWaiting | kCFRunLoopAfterWaiting. As of now, this observer unconditionally context switches back out of the coroutine, punting all the way back to the usual MacOS event loop code -- even during a resize. Concept: proved.

Next steps:

• As of right now wakeup() can't actually wake up the run loop, which I think will mean adding a timer… but again, this is use case. If the caller needs to return by a specific time, they should be able to indicate that when they enter the event loop, instead of waiting for that time to elapse and then trying to rouse the event loop from slumber. Should I open a new issue for that?
• Tear apart the MacOS event loop and reassemble it out of a chain of generators. There's too many ways events can get queued and buffered and such, and still too much logic in poll_events() and run_forever(), even after this refactoring.
• Optimize the coroutine implementation. Right now, each sendEvent: operation gets its own stack and its own runloop observer, both of which are torn down when it returns. This is wasteful.

[willglynn]

Tear apart the MacOS event loop and reassemble it out of a chain of generators.

That's not exactly what I ended up doing, but oh man does this approach feel better. willglynn@12cd9f3:

poll_events() and run_forever() now delegate event production to get_event(), a new function whose job it is to either a) produce Some(Event) by a Timeout or b) return None. This dramatically simplifies the entire event loop.

MacOS in particular has multiple ways that Events can be produced: wakeup(), AppDelegate callbacks, and a Cocoa event receive -> send -> translate cycle. The first two are handled by an event queue; wakeup() now posts an Event::Awakened to the queue, just like the callbacks.

The CocoaEvent cycle is handled either via the usual blocking [NSApp sendEvent:] call or via a coroutine-based event dispatcher. The coroutine yields back to get_event() on a regular basis, even while it's in the middle of sending events. This allows get_event() to return an event to the calling application even if it is still in the middle of a sendEvent: call.

This is what I mean by a timeout-centric API being fundamental. poll_events() and run_forever() are now:

pub fn poll_events<F>(&mut self, mut callback: F)
    where F: FnMut(Event),
{
    // Return as many events as we can without blocking
    while let Some(event) = self.get_event(Timeout::Now) {
        callback(event);
    }
}

pub fn run_forever<F>(&mut self, mut callback: F)
    where F: FnMut(Event) -> ControlFlow
{
    // Get events until we're told to stop
    while let Some(event) = self.get_event(Timeout::Forever) {
        // Send to the app
        let control_flow = callback(event);

        // Do what it says
        match control_flow {
            ControlFlow::Break => break,
            ControlFlow::Continue => (),
        }
    }
}

So far I've implemented only Timeout::Now and Timeout::Forever, but Timeout::Deadline would be a natural and useful extension.

[willglynn]

As of right now wakeup() can't actually wake up the run loop, which I think will mean adding a timer…

Per the earlier changeset, wakeup() actually enqueues an event, so now it's enqueue_event() that's responsible for attempting to wake the runloop.

willglynn@1ffc1bf adds a shared CFRunLoopTimer which can be trigger()ed from any thread. The timer callback does nothing by default, but when paired with --feature context, it yields from event sending coroutine back to get_event().

Experimentation suggested that the timer callback alone was insufficient, as was a timer + runloop observer, but the combination of timer, runloop observer, and explicitly waking the event loop seems at least minimally passable. My crude instrumentation shows enqueue_event() -> get_event() latencies reliably in the 4-6 ms range during a resize. That still seems high, but now AppDelegate callbacks and wakeup() calls are actually getting delivered during a resize, instead of blocking forever until the user stops dragging.

[willglynn]

From #231, @mitchmindtree said:

@willglynn I just tested the window.rs example with your coroutines branch on my macOS machine and it has re-introduced this problem now that you have re-implemented run_forever in terms of get_event.

Are you running with --features context? I'm able to receive resize events during a resize.

[mitchmindtree]

Ah my mistake, I did not! I just ran the window.rs example briefly again, this time with --features "context" and it does seem to emit the events correctly - nice work!

I just had a brief read through the diff for your coroutines branch. I've never done any stack switching before and so I'm yet to wrap my head around it fully (I think reading through this context crate might help), but this solution seems promising.

My crude instrumentation shows enqueue_event() -> get_event() latencies reliably in the 4-6 ms range during a resize

Would you mind elaborating on this a little more? Where is this delay coming from?

[willglynn]

Would you mind elaborating on this a little more? Where is this delay coming from?

Short answer: I don't know.

My coroutines branch adds callbacks on:

• Runloop before sleep
• Runloop after wake
• Timer fired

When triggered from the coroutine, these callbacks yield back to the "normal" stack frame, which jumps back to get_event(). Once there, we check the event queue for pending events, check the timeout to see if we need to return, and eventually yield back to the coroutine -- which from the coroutine's perspective means we finish executing the callback.

The runloop observer keeps us from blocking indefinitely, and the timer is something we can trigger across threads, so we have all the pieces.

Shared::enqueue_event() is thread safe and intended to be invoked from anywhere. It adds the event to the queue (which get_event() will check), and then it needs to get back to get_event(), so it tells the timer to fire right now and calls CFRunLoopWakeUp(CFRunLoopGetMain()). We're then at the mercy of the usual event loop, waiting for it to reach one of our callbacks and yield.

That said, I realized something else: in the case where we're trying to post an event from inside the coroutine -- say because the resize event loop triggered our NSWindowDelegate callback -- we can yield back to get_event() directly. There's no reason to wait for anything at all.

This totally works, and it drops the event delivery time to 11-14 µs using the same unscientific measuring protocol.

[willglynn]

I just had a brief read through the diff for your coroutines branch. I've never done any stack switching before and so I'm yet to warp my head around it fully (I think reading through this context crate might help), but this solution seems promising.

The code in my branch isn't at all a clear introduction to the concept, but it's what I produced after a series of incremental refactorings. My plan for the last action item ("optimization") involves moving the EventLoop boundary even further, and it's easier to explain, so I'll do that now.

The problem

A basic Cocoa runloop looks like:

loop {
    event = receive_event_from_cocoa(distant_future);
    forward_event_to_cocoa(event);
    match event {
        appkit::NSLeftMouseDown => {
            emit(WindowEvent::MouseInput{ … });
        },
        appkit::NSLeftMouseUp => {
            emit(WindowEvent::MouseInput{ … });
        },
        …
    }
}

extern fn resize_callback() {
    emit(WindowEvent::Resized{ … });
}

We need to map this to winit's poll_events() API, in which the user owns the loop {} and in which we need to return promptly. We can call receive_event_from_cocoa() with a deadline that's already in the past, and if we don't get any events, we're done. Sketch:

fn poll_events(callback) {
    while let Some(event) = dequeue_event_nonblocking() {
        callback(event);
    }
    
    global_callback = Some(callback);
    
    while let Some(event) = receive_event_from_cocoa(distant_past) {
        forward_event_to_cocoa(event);
        match event {
            appkit::NSLeftMouseDown => {
                callback(WindowEvent::MouseInput{ … });
            },
            appkit::NSLeftMouseUp => {
                callback(WindowEvent::MouseInput{ … });
            },
            …
        }
    }
    
    global_callback = None;
}

extern fn resize_callback() {
    if let Some(callback) = global_callback {
        callback(WindowEvent::Resized{ … });
    } else {
        enqueue_event(WindowEvent::Resized{ … });
    }
}

This is roughly what master does now. The problem: forward_event_to_cocoa(event) delegates to other components, at least one of which can enter its own internal runloop. This is fine for normal Cocoa apps, since the run inside the loop {} -- but winit runs inside its caller's loop, so entering an internal loop is bad.

The call stack at this point looks like:

poll_events()
  forward_event_to_cocoa()
    -[NSApplication(NSEvent) sendEvent:]
      -[NSWindow(NSEventRouting) sendEvent:]
        -[NSWindow(NSEventRouting) _reallySendEvent:isDelayedEvent:]
          -[NSWindow(NSEventRouting) _handleMouseDownEvent:isDelayedEvent:]
            -[NSThemeFrame mouseDown:]
              -[NSThemeFrame handleMouseDown:]
                -[NSTitledFrame attemptResizeWithEvent:]
                  -[NSWindow(NSWindowResizing) _resizeWithEvent:]
                    -[NSView _getNextResizeEventFromMask:invalidatingLiveResizeCacheIfNecessary:]
                      <internal event loop>
                        resize_callback()

We can get callbacks from within the internal event loop, but we can't return from forward_event_to_cocoa(), and therefore we can't return poll_events(), which means poll_events() will block until that inner loop returns.

The fix that isn't

Okay, so let's split this into two threads: one for the caller's code, one for Cocoa runloop.

fn poll_events(callback) {
    while let Some(event) = dequeue_event_nonblocking() {
        callback(event)
    }
}

fn runloop() {
    loop {
        event = receive_event_from_cocoa(distant_future);
        forward_event_to_cocoa(event);
        match event {
            appkit::NSLeftMouseDown => {
                enqueue_event(WindowEvent::MouseInput{ … });
            },
            appkit::NSLeftMouseUp => {
                enqueue_event(WindowEvent::MouseInput{ … });
            },
            …
        }
    }
}

extern fn resize_callback() {
    enqueue_event(WindowEvent::Resized{ … });
}

thread::spawn(|| { runloop() });

Brilliant. Now it doesn't matter if we're in the outer loop or an inner loop, since the only way runloop() communicates with EventLoops::poll_events() is via a queue or channel or whatever. poll_events() will never block, and we're done.

Problem: receive_event_from_cocoa() and forward_event_to_cocoa() can't be called anywhere but on the main thread, which belongs to the winit's caller, which has its own loop which we don't control.

winit has no choice but to execute a runloop because forward_event_to_cocoa() can enter an internal runloop… but also winit's API puts the user in charge of the runloop and assumes that's the only one. That's a big conflict.

The fix that is

The realization is that while we do need two separate threads of execution -- one of them needs to stay in a loop while the other one returns from a function -- we do not need two separate OS threads. We need only the ability to enter and leave runloop()'s call stack at will, which is something we can achieve.

fn poll_events() {
    while let Some(event) = yield_to_runloop(Timeout::Now) {
        emit(event)
    }
}

static timeout: Timeout;

fn runloop() {
    timeout = yield_to_caller(None);
    
    loop {
        event = receive_event(timeout);
        forward_event(event);
        match event {
            appkit::NSLeftMouseDown => {
                timeout = yield_to_caller(WindowEvent::MouseInput{ … });
            },
            appkit::NSLeftMouseUp => {
                timeout = yield_to_caller(WindowEvent::MouseInput{ … });
            },
            …
        }
    }
}

fn resize_callback() {
    yield_to_caller(WindowEvent::Resized{ … })
}

When poll_events() wants to give control to the runloop, it calls yield_to_runloop(Timeout), which transfers execution into that context. Every time the runloop gets an event, it calls yield_to_caller(Some(Event)). If the runloop detects that the timeout is elapsed, it calls yield_to_caller(None).

At the machine level, this is a user-mode context switch -- context restores the processor state as of the moment the callee was last suspended, while also providing callee with the processor state of the caller as of the moment it transferred execution. This lets the callee resume() its caller, and lets the caller resume() the callee, ad infinitum, cooperatively sharing execution of a single OS thread as coroutines.

The magic is that the runloop() coroutine has a separate call stack, which means it can suspend itself and context switch back to e.g. poll_events() from anywhere. This is the key.

If resize_callback() gets invoked, it can yield back to poll_events() and let the user handle the resize event, even if it's deep inside Cocoa components. The next time the user calls poll_events(), execution resumes inside resize_callback() exactly where it left off. If the runloop learns that forward_event() entered an internal loop due to -[NSWindow(NSWindowResizing) _resizeWithEvent:] and gets a callback that the thread is about to block, it can yield back to poll_events() without returning from _resizeWithEvent:.

Right now in my coroutines branch, message sending lives in a coroutine, which we create and destroy for each message. I plan to push the entire Cocoa runloop into a single long-lived coroutine: stream of Timeouts come in, stream of Option<Event>s come out. This will be simpler, cleaner, and more efficient than the way this branch uses coroutines now.

[tomaka]

So you're context switching by using inline assembly? That looks like dark magic.

[willglynn]

Well… I'm context switching using context, which context switches using boost.context, which context switches using inline assembly. If you'd prefer, I could trade this out for an implementation that FFIs to swapcontext(), which was defined by POSIX and is provided by the MacOS X libc.

(Edit: LLVM is getting coroutine intrinsics, so it stands to reason we'll get them in Rust eventually.)

As I see it, the facts are that a) the MacOS platform has unavoidable loops and b) the winit API assumes it doesn't. The options are:

1. Use coroutines to marry both APIs,
2. Move winit to the bottom of the call stack, drop poll_events(), drop run_forever()'s ability to return, and make the user do everything inside a callback, or
3. Accept that poll_events() can sometimes block forever even though it's not supposed to

[jwilm]

I'm context switching using context, which context switches using boost.context

There's also libfringe which provides this without the boost dependency. libfringe is of course implemented using inline assembly as well.

In any case, the coroutine solution seems rather clever for the Cocoa run loop problem. It would be nice to not worry about macOS-specific details and have the control flow friendly event fetching API.