Enable CB for vlms with multiple images and multiple prompts#583

Merged

quic-hemagnih merged 24 commits intoquic:mainfrom

quic-mamta:cb_vlms

Nov 4, 2025

Contributor

quic-mamta commented Oct 6, 2025

No description provided.

quic-mamta requested review from ochougul, quic-amitraj, quic-hemagnih and quic-rishinr as code owners

October 6, 2025 06:18

quic-mamta marked this pull request as draft

October 6, 2025 06:19

quic-mamta requested review from asmigosw and mohiso22

October 6, 2025 06:19

quic-mamta self-assigned this

quic-mamta force-pushed the cb_vlms branch from 804d672 to 38e777a Compare

October 7, 2025 11:29

quic-hemagnih mentioned this pull request

Llama4 VLM Continuous Batching Support #510

Closed

quic-rishinr mentioned this pull request

Llama4 VLM Continuous Batching Support #559

Closed

quic-rishinr marked this pull request as ready for review

October 29, 2025 10:10

quic-rishinr force-pushed the cb_vlms branch 2 times, most recently from 8dd6395 to b41bd72 Compare

October 31, 2025 18:28

mamtsing and others added 16 commits

November 3, 2025 10:33


          Enable CB for vlms with multiple images and multiple prompts

1ee373e

Signed-off-by: Mamta Singh <mamtsing@qti.qualcomm.com>
Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>


          update text_generation_interface

e9cf657

Signed-off-by: Mamta Singh <mamtsing@qti.qualcomm.com>
Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>


          Updated text_generation to run CB for VLMs

f62f71e

Signed-off-by: Asmita Goswami <asmigosw@qti.qualcomm.com>
Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>


          Ruff format

39f5574

Signed-off-by: Asmita Goswami <asmigosw@qti.qualcomm.com>
Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>


          Updated qwen2_5 modelling for CB support

ecd5905

Signed-off-by: Asmita Goswami <asmigosw@qti.qualcomm.com>
Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>


          Updated qwen2_5 modelling for CB support

3d9cd49

Signed-off-by: Asmita Goswami <asmigosw@qti.qualcomm.com>
Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>


          Passed image queue in decode CB

165c8fb

Signed-off-by: Asmita Goswami <asmigosw@qti.qualcomm.com>
Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>


          Ruff format

cf2d4fb

Signed-off-by: Asmita Goswami <asmigosw@qti.qualcomm.com>
Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>


          refactored the code

d97bda9

Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>


          Qwen2.5vl CB Update

e663cd6

Signed-off-by: Mohit Soni <mohisoni@qti.qualcomm.com>
Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>


          Lint fix and code cleaning

799af59

Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>


          fix for fbs >1

621e3a8

Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>


          Updated cloud_ai_100_exec_kv call in modelling_auto

59dff46

Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>


          Removed redundant vision execution and refactoring

fc69e3a

Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>


          nit: update QBlocking to LM Attention in Qwen2.5VL

275d4cd

Signed-off-by: vbaddi <quic_vbaddi@quicinc.com>
Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>


          nit: update readme for qblocking in example file and lint/format checks

e8253ff

Signed-off-by: vbaddi <quic_vbaddi@quicinc.com>
Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>

vbaddi and others added 5 commits

November 3, 2025 10:33


          nit: lint/format checks

4d9afe2

Signed-off-by: vbaddi <quic_vbaddi@quicinc.com>
Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>


          CI failure fix

6b079fe

Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>


          qwen2_5_vl inference changes

64dffdd

Signed-off-by: Mohit Soni <mohisoni@qti.qualcomm.com>
Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>


          Removed CB regard kw args for functionin non CB models

ca0cc03

Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>


          Added caching for vision outputs

b0ee5a4

Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>

quic-rishinr force-pushed the cb_vlms branch from b41bd72 to b0ee5a4 Compare

November 3, 2025 05:03

quic-dhirajku reviewed

View reviewed changes

QEfficient/generation/embedding_handler.py


          Added change for poping prompt while running continuous batching

a0a80c0

Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>

quic-amitraj requested changes

View reviewed changes

QEfficient/transformers/models/modeling_auto.py Outdated

QEfficient/transformers/models/modeling_auto.py

QEfficient/transformers/models/modeling_auto.py

QEfficient/transformers/models/modeling_auto.py

quic-amitraj reviewed

View reviewed changes

QEfficient/generation/vlm_generation.py


          Adding multi_frame modeling changes and some fix

Signed-off-by: Mohit Soni <mohisoni@qti.qualcomm.com>

quic-amitraj requested changes

View reviewed changes

QEfficient/generation/vlm_generation.py Outdated

QEfficient/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py


          Adressed review comments

875c4d2

Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>

quic-hemagnih approved these changes

View reviewed changes

QEfficient/generation/cloud_infer.py

-                      del self.execObj
-                      self.program.deactivate()
+                      if self.is_active:
+                          del self.execObj

Contributor

quic-hemagnih Nov 4, 2025

NIT pick, we should always have a cleanup method declared for execObj, else it may lead to silent memory leaks. Just deleting the object may lead to hidden memory leaks.

quic-amitraj approved these changes

View reviewed changes

Contributor

quic-amitraj left a comment

LGTM

asmigosw approved these changes

View reviewed changes

Contributor

asmigosw left a comment

LGTM

quic-hemagnih merged commit 7e8838f into quic:main

5 checks passed

abhishek-singh591 pushed a commit to abhishek-singh591/quic_abhishek that referenced this pull request


          Enable CB for vlms with multiple images and multiple prompts (quic#583)

f82d563

Signed-off-by: Mamta Singh <mamtsing@qti.qualcomm.com>
Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>
Signed-off-by: Asmita Goswami <asmigosw@qti.qualcomm.com>
Signed-off-by: Mohit Soni <mohisoni@qti.qualcomm.com>
Signed-off-by: vbaddi <quic_vbaddi@quicinc.com>
Co-authored-by: Mamta Singh <mamtsing@qti.qualcomm.com>
Co-authored-by: Asmita Goswami <asmigosw@qti.qualcomm.com>
Co-authored-by: Rishin Raj <rishinr@qti.qualcomm.com>
Co-authored-by: Mohit Soni <mohisoni@qti.qualcomm.com>
Co-authored-by: Vinayak Baddi <vbaddi@qti.qualcomm.com>

vjanfaza pushed a commit to vjanfaza/efficient-transformers that referenced this pull request


          Enable CB for vlms with multiple images and multiple prompts (quic#583)

20f1c71

Signed-off-by: Mamta Singh <mamtsing@qti.qualcomm.com>
Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>
Signed-off-by: Asmita Goswami <asmigosw@qti.qualcomm.com>
Signed-off-by: Mohit Soni <mohisoni@qti.qualcomm.com>
Signed-off-by: vbaddi <quic_vbaddi@quicinc.com>
Co-authored-by: Mamta Singh <mamtsing@qti.qualcomm.com>
Co-authored-by: Asmita Goswami <asmigosw@qti.qualcomm.com>
Co-authored-by: Rishin Raj <rishinr@qti.qualcomm.com>
Co-authored-by: Mohit Soni <mohisoni@qti.qualcomm.com>
Co-authored-by: Vinayak Baddi <vbaddi@qti.qualcomm.com>
Signed-off-by: Vahid Janfaza <vjanfaza@qti.qualcomm.com>

qraniumcitest mentioned this pull request

Qgenie Prompt testing qraniumcitest/efficient-transformers#46

Closed

abhishek-singh591 pushed a commit to quic-rishinr/efficient-transformers that referenced this pull request


          Enable CB for vlms with multiple images and multiple prompts (quic#583)

Signed-off-by: Mamta Singh <mamtsing@qti.qualcomm.com>
Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>
Signed-off-by: Asmita Goswami <asmigosw@qti.qualcomm.com>
Signed-off-by: Mohit Soni <mohisoni@qti.qualcomm.com>
Signed-off-by: vbaddi <quic_vbaddi@quicinc.com>
Co-authored-by: Mamta Singh <mamtsing@qti.qualcomm.com>
Co-authored-by: Asmita Goswami <asmigosw@qti.qualcomm.com>
Co-authored-by: Rishin Raj <rishinr@qti.qualcomm.com>
Co-authored-by: Mohit Soni <mohisoni@qti.qualcomm.com>
Co-authored-by: Vinayak Baddi <vbaddi@qti.qualcomm.com>
Signed-off-by: abhishek-singh591 <sabhis@qti.qualcomm.com>

quic-dhirajku pushed a commit to quic-dhirajku/efficient-transformers that referenced this pull request


          Enable CB for vlms with multiple images and multiple prompts (quic#583)

e8cc0f7

Signed-off-by: Mamta Singh <mamtsing@qti.qualcomm.com>
Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>
Signed-off-by: Asmita Goswami <asmigosw@qti.qualcomm.com>
Signed-off-by: Mohit Soni <mohisoni@qti.qualcomm.com>
Signed-off-by: vbaddi <quic_vbaddi@quicinc.com>
Co-authored-by: Mamta Singh <mamtsing@qti.qualcomm.com>
Co-authored-by: Asmita Goswami <asmigosw@qti.qualcomm.com>
Co-authored-by: Rishin Raj <rishinr@qti.qualcomm.com>
Co-authored-by: Mohit Soni <mohisoni@qti.qualcomm.com>
Co-authored-by: Vinayak Baddi <vbaddi@qti.qualcomm.com>
Signed-off-by: Dhiraj Kumar Sah <dhirajku@qti.qualcomm.com>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

quic-dhirajku quic-dhirajku left review comments

quic-amitraj quic-amitraj approved these changes

quic-hemagnih quic-hemagnih approved these changes

quic-rishinr Awaiting requested review from quic-rishinr

ochougul Awaiting requested review from ochougul

mohiso22 Awaiting requested review from mohiso22

+1 more reviewer

asmigosw asmigosw approved these changes

Labels

None yet