Skip to content

Conversation

@DrRyanHuang
Copy link
Collaborator

@DrRyanHuang DrRyanHuang commented Aug 28, 2025

本 PR 依赖 Paddle 主框架的两个PR:


#3302 添加了 append_attention_with_output 但是开启后存在打断,本PR消除 full_cuda_graph=false 时的打断

动态图下运行的 cpp_extensions,都是不需要 key_cache_outvalue_cache_out
本PR移除自定义算子注册的 key_cache_outvalue_cache_out,与动态图对齐

另外静态图没有 place,SOT转静的时候会打断:

90afbd244795e119d2a92ca04622f8bb

故而移除 .to(qkv.place)

PS: 目前CUDAGraph + 子图切分的脚本:

source /workspace/env/.py3.10/bin/activate
export MODEL=/ssd1/EB_MODELS/ERNIE-4.5-0.3B-Paddle
rm -rf log/*

export http_proxy=xxx.com:xx
export https_proxy=xxx.com:xx
export no_proxy=xxxx

export FLAGS_cuda_graph_blacklist="custom_op.static_op_append_attention_with_output_"
export PYTHONPATH=/workspace/FastDeploy:/workspace/Paddle/build/python:$PYTHONPATH
export CUDA_VISIBLE_DEVICES=3
export PYTHON_EXCUTOR=/workspace/env/.py3.10/bin/python
export PORT=9905

python -m fastdeploy.entrypoints.openai.api_server \
  --model $MODEL \
  --metrics-port 9717 \
  --port 9718 \
  --engine-worker-queue-port 9719 \
  --tensor-parallel-size 1 \
  --max-model-len 32768 \
  --max-num-seqs 128 \
  --quantization wint4 \
  --graph-optimization-config '{"graph_opt_level": 1, "use_cudagraph": true, "full_cuda_graph": false}' \

cc @SigureMo @zyfncg @gongshaotian

@paddle-bot
Copy link

paddle-bot bot commented Aug 28, 2025

Thanks for your contribution!

gongshaotian
gongshaotian previously approved these changes Aug 28, 2025
Copy link
Collaborator

@gongshaotian gongshaotian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@codecov-commenter
Copy link

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (develop@808b548). Learn more about missing BASE report.

Additional details and impacted files
@@            Coverage Diff            @@
##             develop   #3694   +/-   ##
=========================================
  Coverage           ?   0.00%           
=========================================
  Files              ?       1           
  Lines              ?       3           
  Branches           ?       0           
=========================================
  Hits               ?       0           
  Misses             ?       3           
  Partials           ?       0           
Flag Coverage Δ
diff 0.00% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@DrRyanHuang DrRyanHuang changed the title [SOT] Eliminate BreakGraph caused by #3302 [SOT] Eliminate BreakGraph caused by #3302 && update CustomOp Sep 4, 2025
@DrRyanHuang DrRyanHuang changed the title [SOT] Eliminate BreakGraph caused by #3302 && update CustomOp [SOT][Cudagraph] Remove BreakGraph of #3302 && update CustomOp Sep 4, 2025
paddle::Optional("q_norm_weight"),
paddle::Optional("k_norm_weight")})
.Outputs({"fmha_out", "key_cache_out", "value_cache_out"})
.SetInplaceMap({{"key_cache", "key_cache_out"},
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里的 inplace map 删掉是符合预期的么?

Copy link
Collaborator Author

@DrRyanHuang DrRyanHuang Sep 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

key_cachevalue_cache 都是没有用到的输出,可以删除

这个是 append_attention 的OP注册,不是 append_attention_with_output 的注册,append_attention 不是 inplace 的输出,是内部创建的

Copy link
Member

@SigureMo SigureMo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTMeow 🐾

@DrRyanHuang
Copy link
Collaborator Author

DrRyanHuang commented Oct 13, 2025

已添加单测


目前 fastdeploy/model_executor/layers/attention/append_attn_backend.py 这几行覆盖不到
是因为 append_attention_with_output 目前测试不到,开启 full_cuda_graph=false 就能测到了,后续CI添加

  --graph-optimization-config '{"graph_opt_level": 1, "use_cudagraph": true, "full_cuda_graph": false}'

cc @gongshaotian @SigureMo

gongshaotian
gongshaotian previously approved these changes Oct 13, 2025
Copy link
Collaborator

@gongshaotian gongshaotian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Jiang-Jia-Jun Jiang-Jia-Jun merged commit 49cea8f into PaddlePaddle:develop Oct 17, 2025
32 of 38 checks passed
@DrRyanHuang DrRyanHuang deleted the append_attention branch October 17, 2025 02:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants