Add Lita, Vila and Vita TRTLLM export by xuanzic · Pull Request #9734 · NVIDIA-NeMo/NeMo

xuanzic · 2024-07-15T17:19:25Z

What does this PR do ?

modify the neva_model to remove hardcoded 'vision_encoder" name.
add trtllm export support for lita, vila and vita .nemo model.
add run example for running lita, vila and vita inference.
add triton deployment for these models.

Collection: [Note which collection this PR will affect]

Changelog

Add specific line by line info of high level changes in this PR.

Usage

You can potentially add a usage example below

# Add a code snippet demonstrating how to use this

GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

Before your PR is "Ready for review"

Pre checks:

Make sure you read and followed Contributor guidelines
Did you write any new necessary tests?
Did you add or update any necessary documentation?
Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
- Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

New Feature
Bugfix
Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

Related to # (issue)

Signed-off-by: Vivian Chen <xuanzic@example.com>

Signed-off-by: Vivian Chen <xuanzic@nvidia.com>

Signed-off-by: Vivian Chen <xuanzic@example.com>

examples/multimodal/multimodal_llm/neva/neva_export.py

Signed-off-by: Vivian Chen <xuanzic@example.com>

Signed-off-by: xuanzic <xuanzic@users.noreply.github.com>

nemo/export/multimodal/run.py

+        return image
+
+    def process_image(self, image_file, image_processor, nemo_config, image_folder):
+        image_processor = image_processor


nemo/export/multimodal/run.py

+from torch.nn import functional as F
 from torchvision import transforms
-from transformers import CLIPImageProcessor
+from transformers import AutoModel, AutoProcessor, CLIPImageProcessor


nemo/export/multimodal/build.py

+        vision_max_batch_size,
        dtype,
        image_size=image_size,
+        num_frames=lita_num_frames if model_type == "lita" or model_type == 'vita' else None,


nemo/export/multimodal/run.py

-        else:
-            image = image.expand(batch_size, -1, -1, -1).contiguous()
+        if self.model_type not in ['vila', 'lita', 'vita']:
+            if image.dim() == 5:


Signed-off-by: Vivian Chen <xuanzic@example.com>

Signed-off-by: xuanzic <xuanzic@users.noreply.github.com>

nemo/export/multimodal/run.py

+        elif isinstance(video_path, np.ndarray):
+            frames = torch.tensor(video_path, dtype=torch.float32)
+
+        return self.preprocess_frames(frames, config, processor)


nemo/export/multimodal/run.py

+                .unsqueeze(0)
+                .to(self.device, dtype=torch.bfloat16)
+            )
+        return image


Signed-off-by: Vivian Chen <xuanzic@example.com>

meatybobby

LGTM

* modify vision encoder config Signed-off-by: Vivian Chen <xuanzic@example.com> * add lita, vila engine build support and fix export api bugs Signed-off-by: Vivian Chen <xuanzic@nvidia.com> * add run example for vila, lita and vita Signed-off-by: Vivian Chen <xuanzic@example.com> * couple of changes for exporter Signed-off-by: Vivian Chen <xuanzic@example.com> * Apply isort and black reformatting Signed-off-by: xuanzic <xuanzic@users.noreply.github.com> * address code scanning issues Signed-off-by: Vivian Chen <xuanzic@example.com> * add triton deployment for lita/vila/vita Signed-off-by: Vivian Chen <xuanzic@example.com> * Apply isort and black reformatting Signed-off-by: xuanzic <xuanzic@users.noreply.github.com> * fix code scan Signed-off-by: Vivian Chen <xuanzic@example.com> --------- Signed-off-by: Vivian Chen <xuanzic@example.com> Signed-off-by: Vivian Chen <xuanzic@nvidia.com> Signed-off-by: xuanzic <xuanzic@users.noreply.github.com> Co-authored-by: Vivian Chen <xuanzic@example.com> Co-authored-by: xuanzic <xuanzic@users.noreply.github.com>

* modify vision encoder config Signed-off-by: Vivian Chen <xuanzic@example.com> * add lita, vila engine build support and fix export api bugs Signed-off-by: Vivian Chen <xuanzic@nvidia.com> * add run example for vila, lita and vita Signed-off-by: Vivian Chen <xuanzic@example.com> * couple of changes for exporter Signed-off-by: Vivian Chen <xuanzic@example.com> * Apply isort and black reformatting Signed-off-by: xuanzic <xuanzic@users.noreply.github.com> * address code scanning issues Signed-off-by: Vivian Chen <xuanzic@example.com> * add triton deployment for lita/vila/vita Signed-off-by: Vivian Chen <xuanzic@example.com> * Apply isort and black reformatting Signed-off-by: xuanzic <xuanzic@users.noreply.github.com> * fix code scan Signed-off-by: Vivian Chen <xuanzic@example.com> --------- Signed-off-by: Vivian Chen <xuanzic@example.com> Signed-off-by: Vivian Chen <xuanzic@nvidia.com> Signed-off-by: xuanzic <xuanzic@users.noreply.github.com> Co-authored-by: Vivian Chen <xuanzic@example.com> Co-authored-by: xuanzic <xuanzic@users.noreply.github.com> Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* modify vision encoder config Signed-off-by: Vivian Chen <xuanzic@example.com> * add lita, vila engine build support and fix export api bugs Signed-off-by: Vivian Chen <xuanzic@nvidia.com> * add run example for vila, lita and vita Signed-off-by: Vivian Chen <xuanzic@example.com> * couple of changes for exporter Signed-off-by: Vivian Chen <xuanzic@example.com> * Apply isort and black reformatting Signed-off-by: xuanzic <xuanzic@users.noreply.github.com> * address code scanning issues Signed-off-by: Vivian Chen <xuanzic@example.com> * add triton deployment for lita/vila/vita Signed-off-by: Vivian Chen <xuanzic@example.com> * Apply isort and black reformatting Signed-off-by: xuanzic <xuanzic@users.noreply.github.com> * fix code scan Signed-off-by: Vivian Chen <xuanzic@example.com> --------- Signed-off-by: Vivian Chen <xuanzic@example.com> Signed-off-by: Vivian Chen <xuanzic@nvidia.com> Signed-off-by: xuanzic <xuanzic@users.noreply.github.com> Co-authored-by: Vivian Chen <xuanzic@example.com> Co-authored-by: xuanzic <xuanzic@users.noreply.github.com>

* modify vision encoder config Signed-off-by: Vivian Chen <xuanzic@example.com> * add lita, vila engine build support and fix export api bugs Signed-off-by: Vivian Chen <xuanzic@nvidia.com> * add run example for vila, lita and vita Signed-off-by: Vivian Chen <xuanzic@example.com> * couple of changes for exporter Signed-off-by: Vivian Chen <xuanzic@example.com> * Apply isort and black reformatting Signed-off-by: xuanzic <xuanzic@users.noreply.github.com> * address code scanning issues Signed-off-by: Vivian Chen <xuanzic@example.com> * add triton deployment for lita/vila/vita Signed-off-by: Vivian Chen <xuanzic@example.com> * Apply isort and black reformatting Signed-off-by: xuanzic <xuanzic@users.noreply.github.com> * fix code scan Signed-off-by: Vivian Chen <xuanzic@example.com> --------- Signed-off-by: Vivian Chen <xuanzic@example.com> Signed-off-by: Vivian Chen <xuanzic@nvidia.com> Signed-off-by: xuanzic <xuanzic@users.noreply.github.com> Co-authored-by: Vivian Chen <xuanzic@example.com> Co-authored-by: xuanzic <xuanzic@users.noreply.github.com> Signed-off-by: Hainan Xu <hainanx@nvidia.com>

* modify vision encoder config Signed-off-by: Vivian Chen <xuanzic@example.com> * add lita, vila engine build support and fix export api bugs Signed-off-by: Vivian Chen <xuanzic@nvidia.com> * add run example for vila, lita and vita Signed-off-by: Vivian Chen <xuanzic@example.com> * couple of changes for exporter Signed-off-by: Vivian Chen <xuanzic@example.com> * Apply isort and black reformatting Signed-off-by: xuanzic <xuanzic@users.noreply.github.com> * address code scanning issues Signed-off-by: Vivian Chen <xuanzic@example.com> * add triton deployment for lita/vila/vita Signed-off-by: Vivian Chen <xuanzic@example.com> * Apply isort and black reformatting Signed-off-by: xuanzic <xuanzic@users.noreply.github.com> * fix code scan Signed-off-by: Vivian Chen <xuanzic@example.com> --------- Signed-off-by: Vivian Chen <xuanzic@example.com> Signed-off-by: Vivian Chen <xuanzic@nvidia.com> Signed-off-by: xuanzic <xuanzic@users.noreply.github.com> Co-authored-by: Vivian Chen <xuanzic@example.com> Co-authored-by: xuanzic <xuanzic@users.noreply.github.com>

Vivian Chen and others added 2 commits July 11, 2024 16:42

modify vision encoder config

8a67982

Signed-off-by: Vivian Chen <xuanzic@example.com>

add lita, vila engine build support and fix export api bugs

3e65e6a

Signed-off-by: Vivian Chen <xuanzic@nvidia.com>

github-actions bot added the Multi Modal label Jul 15, 2024

xuanzic changed the title ~~Add Lita, Vila and Vita TRTLLM export~~ [DRAFT] Add Lita, Vila and Vita TRTLLM export Jul 15, 2024

xuanzic marked this pull request as draft July 15, 2024 17:27

add run example for vila, lita and vita

d87455e

Signed-off-by: Vivian Chen <xuanzic@example.com>

meatybobby reviewed Jul 16, 2024

View reviewed changes

examples/multimodal/multimodal_llm/neva/neva_export.py Show resolved Hide resolved

couple of changes for exporter

df4228f

Signed-off-by: Vivian Chen <xuanzic@example.com>

xuanzic changed the title ~~[DRAFT] Add Lita, Vila and Vita TRTLLM export~~ Add Lita, Vila and Vita TRTLLM export Jul 17, 2024

xuanzic marked this pull request as ready for review July 17, 2024 03:21

xuanzic and others added 2 commits July 16, 2024 20:21

Merge branch 'main' into vchen/vila-infer

7d17754

Apply isort and black reformatting

d1f05fd

Signed-off-by: xuanzic <xuanzic@users.noreply.github.com>

github-advanced-security bot found potential problems Jul 17, 2024

View reviewed changes

Vivian Chen and others added 3 commits July 17, 2024 04:39

address code scanning issues

f33bf64

Signed-off-by: Vivian Chen <xuanzic@example.com>

add triton deployment for lita/vila/vita

c0427d9

Signed-off-by: Vivian Chen <xuanzic@example.com>

Apply isort and black reformatting

62b5fa7

Signed-off-by: xuanzic <xuanzic@users.noreply.github.com>

github-advanced-security bot found potential problems Jul 18, 2024

View reviewed changes

fix code scan

a1ea8cc

Signed-off-by: Vivian Chen <xuanzic@example.com>

meatybobby approved these changes Jul 18, 2024

View reviewed changes

meatybobby added the Run CICD label Jul 18, 2024

xuanzic added 2 commits July 18, 2024 08:13

Merge branch 'main' into vchen/vila-infer

e176675

Merge branch 'main' into vchen/vila-infer

ab66b47

meatybobby added Run CICD and removed Run CICD labels Jul 19, 2024

meatybobby merged commit ab8988e into NVIDIA-NeMo:main Jul 19, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Lita, Vila and Vita TRTLLM export#9734

Add Lita, Vila and Vita TRTLLM export#9734
meatybobby merged 12 commits intoNVIDIA-NeMo:mainfrom
xuanzic:vchen/vila-infer

xuanzic commented Jul 15, 2024 •

edited

Loading

Uh oh!

Uh oh!

Check failure

Check notice

Check failure

Check failure

Check failure

Check failure

meatybobby left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Comments

Conversation

xuanzic commented Jul 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do ?

Changelog

Usage

GitHub Actions CI

Before your PR is "Ready for review"

Who can review?

Additional Information

Uh oh!

Uh oh!

Check failure

Check notice

Check failure

Check failure

Check failure

Check failure

meatybobby left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Comments

xuanzic commented Jul 15, 2024 •

edited

Loading