Skip to content

[Feature] support async download features#5003

Merged
Jiang-Jia-Jun merged 4 commits intoPaddlePaddle:developfrom
kevincheng2:cp_async_down
Nov 19, 2025
Merged

[Feature] support async download features#5003
Jiang-Jia-Jun merged 4 commits intoPaddlePaddle:developfrom
kevincheng2:cp_async_down

Conversation

@kevincheng2
Copy link
Collaborator

@kevincheng2 kevincheng2 commented Nov 13, 2025

Motivation

support async download features
cp from #4910

  1. 将features的下载从 worker进程转移到 engine进程,避免gpu等待features下载,浪费算力。
  2. engine 调度时增加异步等待机制,未完成异步处理的请求不下发至worker(支持多个异步事件,后续可以扩展其他异步任务)。
  3. 取消features下载检查逻辑,下载发生错误时可以直接返回错误信息。

Modifications

将features下载由worker转移到engine中

Usage or Command

启动时需要增加参数:

  python -m fastdeploy.entrypoints.openai.api_server \
       ...
       --enable-async-download-features

性能测试后,删除该启动参数,改为默认打开

Accuracy Tests

Checklist

  • Add at least a tag in the PR title.
    • Tag list: [[FDConfig],[APIServer],[Engine], [Scheduler], [PD Disaggregation], [Executor], [Graph Optimization], [Speculative Decoding], [RL], [Models], [Quantization], [Loader], [OP], [KVCache], [DataProcessor], [BugFix], [Docs], [CI], [Optimization], [Feature], [Benchmark], [Others], [XPU], [HPU], [GCU], [DCU], [Iluvatar], [Metax]]
    • You can add new tags based on the PR content, but the semantics must be clear.
  • Format your code, run pre-commit before commit.
  • Add unit tests. Please write the reason in this PR if no unit tests.
  • Provide accuracy results.
  • If the current PR is submitting to the release branch, make sure the PR has been submitted to the develop branch, then cherry-pick it to the release branch with the [Cherry-Pick] PR tag.

@paddle-bot
Copy link

paddle-bot bot commented Nov 13, 2025

Thanks for your contribution!

@Jiang-Jia-Jun Jiang-Jia-Jun merged commit 109d48e into PaddlePaddle:develop Nov 19, 2025
21 of 26 checks passed
@kevincheng2 kevincheng2 deleted the cp_async_down branch January 19, 2026 03:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants