test(scheduler): repro signal watcher failure during SIGINT cleanup

[anmonteiro]

• Replace the Unix SIGCHLD wakeup path in the scheduler with a self-pipe notifier.
• Keep child reaping on the scheduler thread, preserving the invariant from Scheduler: fix pid races #13245.
• Install a minimal SIGCHLD handler in C that performs only an async-signal-safe write to a pipe.
• On Unix systems, add notifier thread in process watcher that reads pipe events and emits Job_complete_ready.
• On macOS, fixes a failure mode where interrupted high-concurrency builds could crash with signal watcher received an unexpected signal / Unknown. This was previously thought fixed in fix(signal-watcher): don't assert false on SIGKILL #13370

[rgrinberg]

Could you explain the bug that is being fixed here?

[rgrinberg]

Glancing over the fix, the changes don't really make sense as they essentially introduce a separate signal watching thread for SIGCHLD. Why would we want that unless using Thread.wait_signal is buggy? But if it's buggy, why would we use it for the other signals? In any case, I think you should explain the buggy behavior that you're seeing and what makes you suspect our signal watching thread is the cause.

[anmonteiro]

For sure, perhaps I haven't gotten to the absolute root cause of the issue, but the 1st commit reproduces the issue in our macOS tests.

I will do more digging to see if I can unearth something, but essentially under high process concurrency / cancelation, I observe the assert false case of the signal pattern match just has garbage (e.g. Unknown 123541251) as the signla.

[rgrinberg]

That's concerning, but I'm not sure I'd suspect the issue is in dune. I would suspect there's something wrong with the OCaml runtime if it's returning us a garbage signal.

Why can't we just ignore such junk signals?

[anmonteiro]

we may just be able to ignore the junk signals.

I'll push an alternative fix doing that, instead.

In parallel, I'm trying to track down the specific bug, at this point I agree it's not in Dune.

[rgrinberg]

FWIW, we don't necessarily need to watch signals in a separate thread. If watch signals in the main thread works better on macos, then so be it. But we need to do it for all signals. Moreover, we need to justify why we can't write the signal handlers in OCaml.

[anmonteiro]

I haven't found a good way to write this:

• in pure OCaml -- my guess is that it has to do with the way OCaml runs signal handlers
• with just kqueue on macOS

so far, the only solution that i've seen work well is handling only SIGCHLD via the self-pipe with a sig handler in C stubs, and handling the remaining signals in OCaml like before.

so i'm proceeding with the mitigation in #14047, which at least gets Dune unstuck.