[Tracking] Advancements for P2P

[hendrikmakait]

Bug fixes

• P2P with null partitions fails #8092

Performance

• P2P blows up memory #8015
• P2P DataFrame performance - Disk overhead #7990
• Optional Disk for P2P shuffling #7572
  • [RFC]: Allow in-memory buffering of p2p shuffle results #7618
• P2P shuffle is slow with string dtypes #7880
  • Possibly fixed as of Improved conversion between pyarrow and pandas in P2P shuffling #7896, needs to be evaluated.
• RFC Set priorities for p2p shuffle tasks #7926
• RFC P2P pass arrow tables directly to buffers #7992

Robustness

• P2P Be robust to connection timeouts during shuffle_receive #8011

Diagnostics

• Use metering for P2P shuffling instrumentation #7943

Refactoring

• Restructure P2P code #8098
• Simplify state management in P2P's ShuffleSchedulerPlugin #8018
• Make P2P shuffle extensible #8096

Documentation

• Improve user-facing documentation
• Add documentation for consistency/resilience mechanism
• Add documentation for developing your own P2P-based shuffle
• ...

cuDF compatibility

• Enable basic p2p shuffle for dask-cudf #7743