Skip to content

Conversation

@zeroRains
Copy link
Contributor

@zeroRains zeroRains commented Sep 4, 2025

pcard-71500

问题描述

  1. PD分离+CudaGraph启动服务是, D节点出现了real shape:0 is not in cuda graph capture list的问题。
  2. cache_messager.py 报错such file or directory: '/splitwise_complete_prefilled_layer_4.4'

主要原因:

  1. 在D节点捕获阶段使用了256的batch_size进行捕获,导致后续计算时,实际的input_length计算为0。具体原因见下图
92a552f012b9980cffcfa68e8bf80722
  1. 信号同名了,修改一下信号名字

解决方案:

  1. 一开始计算input_length时向上取整,确保input_length不为0,同时移除与 self.cache_config.kv_cache_ratio相乘的步骤。
  2. 修改信号名称

@paddle-bot
Copy link

paddle-bot bot commented Sep 4, 2025

Thanks for your contribution!

@Jiang-Jia-Jun Jiang-Jia-Jun merged commit d435499 into PaddlePaddle:release/2.2 Sep 8, 2025
14 of 16 checks passed
@zeroRains zeroRains deleted the cp branch September 8, 2025 06:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants