[Roadmap] TransferQueue Q1 roadmap

[0oshowero0]

Feature

• User-defined Metadata: Support put/get operations for user-defined metadata in the TQ controller. @0oshowero0
• StreamingDataLoader: Optimize the user experience. [StreamingDataLoader, 4/N] feat: Introduce sample pre-allocation for dynamic streaming #16 @0oshowero0
• Easy-to-use Backend: Provide backend-specific metadata storage capability, simplifying RDMA transport (eliminate the need to use zmq to fetch some dtype & shape info by users). @tianyi-ge [feat] Support store custom_meta in controller for backend-specific info #5, https://gitcode.com/Ascend/TransferQueue/pull/14

Integrations

verl

tracking issue: verl-project/verl#4880

• Upstream Merge: Merge TQ integration into the verl main codebase [WIP][data] feat: TransferQueue - integrate TransferQueue into main codebase verl-project/verl#4987
• Multi-modal Support: Optimize TQ performance for multi-modal scenarios.
• Router-Replay: Implement support for router-replay transmission with TQ.
• Support next-gen single-controller development

Optimization

• Zero-Copy Serialization: Implement faster, robust zero-copy serialization https://gitcode.com/Ascend/TransferQueue/pull/7, https://gitcode.com/Ascend/TransferQueue/pull/19, https://gitcode.com/Ascend/TransferQueue/pull/22 @mpb159753
• User experience: Optimize initialization of storage manager: [Enhancement] A suggestion regarding StorageManager initialization #6 @dpj135

Backend Supports

• Yuanrong Backend: Optimize backend and provide E2E performance benchmarks on both GPU and NPU. @tianyi-ge

Documentation & Tutorial

• Website: Build a dedicated documentation website.
• API Reference: Generate a comprehensive API list.
• Onboarding: Provide guiding docs to simplify the user onboarding process.
• Tutorial: Add pluggable backend tutorial, demonstrate how to switch between supported backends and illustrate the custom_backend_meta usage. @tianyi-ge

CI/CD

• Unified Workflow: Unify CI experience across GitCode and GitHub (addressing missing tests on GitCode).
• E2E Testing: Enable End-to-End tests (resolve execution issues on GitHub Actions).
• Performance Checkpoint: Add periodic performance testing to detect regressions in new PRs.
• Refactoring: Refactor unit test logic for better maintainability.