Data-plane perf overhaul (+~2× single-stream TCP)

[mmalmi]

Data-plane perf overhaul: off-task encrypt + decrypt, GSO, connected UDP

TL;DR

Moves both AEAD (Authenticated Encryption with Associated Data — ChaCha20-Poly1305 in FIPS, one round per layer per packet) layers plus the sendmsg syscall off the rx_loop task onto a per-shard worker pool, adds per-peer connect(2)-ed UDP with SO_REUSEPORT, and uses Linux UDP GSO (Generic Segmentation Offload — the kernel splits one large super-skb into N on-the-wire datagrams in a single trip through the TX stack) when packets in a batch are uniform-size. GSO is the same kernel primitive WireGuard's in-kernel module and Cloudflare's BoringTun userspace tunnel use to hit 2.5–3.2 Gbps single-stream.

Single TCP stream: ~+1.9× across every static-peer path.

Path	Before this PR (Mbps)	With this PR (Mbps)	Speedup	RTT Δ
A→D	1379	2708	1.96×	+0.12 ms
A→E	1394	2663	1.91×	+0.11 ms
E→A	1406	2624	1.87×	+0.19 ms

testing/static/scripts/bench-multirun.sh, 5 × 15 s × 1 stream medians on a Linux x86_64 docker-bridge mesh (8 vCPU / 24 GiB, kernel 7.0). All CoV < 3 %, 0 outliers (Δ > 20 % from median), 0 % ICMP loss on both branches. The +100–200 µs RTT is the worker queue handoff added by moving AEAD off rx_loop.

Topology

5 nodes on one docker-bridge subnet (testing/static/configs/topologies/mesh.yaml). Static peers::

A: D, E       B: C        C: B, D, E
D: A, C, E    E: A, C, D

Diff shape

30 files, +6173 / -101. ~85 % of new code is in six new files that drop in alongside the existing transport stack:

File	Lines	Role
src/node/encrypt_worker.rs	+2238	FMP+FSP AEAD-seal + sendmmsg / UDP_GSO worker pool
src/node/decrypt_worker.rs	+693	shard-owned receive AEAD-open + replay window
src/transport/udp/peer_drain.rs	+457	per-peer recv drain thread
src/transport/udp/connected_peer.rs	+436	per-peer connect(2)-ed UDP socket
src/perf_profile.rs	+405	optional per-stage timing reporter
src/transport/udp/darwin_sockopts.rs	+197	macOS UDP tuning

The rest is integration glue. All #[cfg(unix)]-gated; Windows continues on the existing tokio-based send/recv.

What's in scope

• Off-task encrypt (encrypt_worker) — std::thread + crossbeam_channel; hash-by-destination dispatch pins a TCP flow to one worker so wire ordering is preserved; per-worker sendmmsg(2) batching up to 32; Linux uses sendmsg(2)+UDP_SEGMENT when packets in a group are uniform-size.
• Off-task decrypt (decrypt_worker) — receive-side mirror. Each shard owns its session's recv cipher + replay window in a thread-local HashMap (no shared RwLock/Mutex). Sessions are handed off at promote_connection and re-registered on K-bit flip / rekey cutover.
• FSP+FMP pipelined send — both AEAD layers seal in-place in the worker on a single wire-buffer allocation; no intermediate inner_plaintext / fsp_payload Vecs.
• Per-peer connected UDP (Linux + macOS) — SO_REUSEPORT so the per-peer connected socket can bind to the same wildcard port the listen socket holds; the worker sends with msg_name = NULL and the kernel uses its cached 5-tuple (skips per-packet route + neighbor lookup). Tick-driven activation, idempotent.
• Receive zero-copy — mem::replace the recvmmsg backing buffer instead of to_vec() per packet; SessionDatagramRef zero-copy view for local delivery; TransportAddr::from_socket_addr collapses two allocs to one.
• rx_loop ordering — fallback drain promoted ahead of packet_rx in the select!; interleaved fallback drain every 32 packets inside the rx burst loop so TCP ACKs don't pile up behind a 256-packet inbound burst.
• Worker pool sizing — both default to num_cpus, overridable via FIPS_ENCRYPT_WORKERS=N / FIPS_DECRYPT_WORKERS=N.
• FIPS_PERF=1 — optional per-stage timing reporter. Off by default, zero overhead when disabled.
• Bench harness (testing/static/scripts/bench-multirun.sh) — N reruns (default 5), median / min / max / CoV % / per-run outlier flag, avg ping RTT, ICMP loss %, TCP retransmit total. Pre-bench peer-convergence check + per-path route verification via stats.bytes_sent deltas — fails fast if traffic exits via a non-static-peer link.

Test plan

• cargo fmt --check clean
• cargo clippy --all-targets --all-features -- -D warnings clean on Linux (what CI runs)
• cargo nextest run --all --profile ci — 1228 passed / 4 skipped / 0 failed
• Session-specific tests 127 / 127, rekey-specific 9 / 9
• bash -n on both modified shell scripts
• CI to confirm macOS + Windows builds

[jmcorgan]

This PR has gone DIRTY against current master. Three landings arrived after it opened: #89 macOS recvmsg_x at 59225cc, #90 rx zero-copy at b1af151, and the coord cache surgical-invalidation merge wave at f51dde6. A rebase onto current master is needed before this can land.

Once you've rebased, I'll do a detailed pass and follow up. Thanks.

[mmalmi]

Rebased onto current master (2b6d402). Status is now MERGEABLE / CLEAN.

Conflicts resolved:

• src/protocol/link.rs — master's rx: avoid copies in receive hot paths #90 already added SessionDatagramRef (b1af151); dropped the PR's redundant duplicate definition, kept master's slightly cleaner SessionDatagram::decode → SessionDatagramRef::decode + into_owned() flow.
• src/node/handlers/forwarding.rs — same situation; kept master's ref-based handle_session_datagram (uses into_owned(); both branches independently arrived at the same zero-copy ref pattern).
• src/transport/mod.rs, src/transport/udp/mod.rs — comment-only conflicts; kept master's comment in TransportAddr::from_socket_addr, kept the PR's mem::replace explanation in the recv loop.

Verified on Linux x86_64 (matches CI target):

• cargo fmt --check clean
• cargo clippy --all-targets --all-features -- -D warnings clean
• cargo nextest run --all --profile ci: 1247 passed / 6 skipped / 0 failed (222s)

Also built on Windows — succeeds, same pre-existing Windows-only dead-code warnings as before the rebase (process_authentic_fmp_plaintext only referenced from cfg(unix) worker code; ESTABLISHED_HEADER_SIZE import unused on non-unix). Not regressions from the rebase, but happy to gate them on the next pass if you'd like.