Add option to convert PyTriton response to OpenAI format by athitten · Pull Request #9726 · NVIDIA-NeMo/NeMo

athitten · 2024-07-14T00:39:18Z

What does this PR do ?

Adds option to convert responses from PyTriton server to OpenAI compatible format.
Also adds triton_request_timeout and openai_format_response as user defined args which are then written to a json file along with triton IP and triton port to be accessed by rest_model_api.py (Deletes the existing nemo/deploy/service/config.json favor of these changes)

Collection: [Note which collection this PR will affect]

Changelog

Add specific line by line info of high level changes in this PR.

Usage

You can potentially add a usage example below

# Add a code snippet demonstrating how to use this

GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

Before your PR is "Ready for review"

Pre checks:

Make sure you read and followed Contributor guidelines
Did you write any new necessary tests?
Did you add or update any necessary documentation?
Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
- Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

New Feature
Bugfix
Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

Related to # (issue)

Signed-off-by: Abhishree <abhishreetm@gmail.com>

Signed-off-by: athitten <athitten@users.noreply.github.com>

athitten · 2024-07-16T23:38:16Z

nemo/deploy/service/rest_model_api.py


+@app.get("/triton_health")
+async def check_triton_health():
+    """


Added a test method check_triton_health for users to verify if Pytriton server is up and running since many times while using REST service, we encounter an error if Pytriton server is not accessible. This test method adds a check for that.

athitten · 2024-07-16T23:43:43Z

scripts/deploy/nlp/deploy_triton.py

    )
+    parser.add_argument(
+        "-trt", "--triton_request_timeout", default=60, type=int, help="Timeout in seconds for Triton server"
+    )


Moved the args triton_request_timeout from nemo/service/deploy/config.json to the argparser of deploy_triton.py so that users can pass it. Also added openai_format_response as an user defined option while running deploy_triton.py with REST service True

Also deleted config.json as triton IP, triton port, triton timeout args are passed by the user while running deploy_triton.py and they are stored in a json file when deploy_triton.py is run which are then accessed in rest_model_api.py via the stored config.json file.

oyilmaz-nvidia · 2024-07-17T16:58:11Z

nemo/deploy/nlp/query_llm.py


                sentences = np.char.decode(output.astype("bytes"), "utf-8")
-                return sentences
+                if openai_format_response:


Do you think we can do this openai response conversion in the here https://github.com/NVIDIA/NeMo/blob/athitten/openai_format_response/nemo/deploy/service/rest_model_api.py#L101 ?

Also, is there any advantage of supporting the openai response type at the class level? Maybe our default should be the openai response format (still not sure though).

@oyilmaz-nvidia yes we can do the openAI response conversion in rest_model_api.py after output is received here. Do you recommend that ?

Also regarding having OpenAI format as the default, do any of the methods/functions in the deploy module or any downstream tasks rely on output being returned directly (non OpenAI format) i.e the way it is now ? I wasn't sure of this so added it as an option to not break anything.

So, this is an end user API and not sure who is using it how. So, adding OpenAI format as option makes sense. Let's leave it as it is.

oyilmaz-nvidia

LGTM.

* Option to convert response to OPenAI format Signed-off-by: Abhishree <abhishreetm@gmail.com> * Add OpenAI response arg and store_args_to_json method Signed-off-by: Abhishree <abhishreetm@gmail.com> * Apply isort and black reformatting Signed-off-by: athitten <athitten@users.noreply.github.com> --------- Signed-off-by: Abhishree <abhishreetm@gmail.com> Signed-off-by: athitten <athitten@users.noreply.github.com> Co-authored-by: athitten <athitten@users.noreply.github.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com>

…#9726) * Option to convert response to OPenAI format Signed-off-by: Abhishree <abhishreetm@gmail.com> * Add OpenAI response arg and store_args_to_json method Signed-off-by: Abhishree <abhishreetm@gmail.com> * Apply isort and black reformatting Signed-off-by: athitten <athitten@users.noreply.github.com> --------- Signed-off-by: Abhishree <abhishreetm@gmail.com> Signed-off-by: athitten <athitten@users.noreply.github.com> Co-authored-by: athitten <athitten@users.noreply.github.com>

* Option to convert response to OPenAI format Signed-off-by: Abhishree <abhishreetm@gmail.com> * Add OpenAI response arg and store_args_to_json method Signed-off-by: Abhishree <abhishreetm@gmail.com> * Apply isort and black reformatting Signed-off-by: athitten <athitten@users.noreply.github.com> --------- Signed-off-by: Abhishree <abhishreetm@gmail.com> Signed-off-by: athitten <athitten@users.noreply.github.com> Co-authored-by: athitten <athitten@users.noreply.github.com> Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* Option to convert response to OPenAI format Signed-off-by: Abhishree <abhishreetm@gmail.com> * Add OpenAI response arg and store_args_to_json method Signed-off-by: Abhishree <abhishreetm@gmail.com> * Apply isort and black reformatting Signed-off-by: athitten <athitten@users.noreply.github.com> --------- Signed-off-by: Abhishree <abhishreetm@gmail.com> Signed-off-by: athitten <athitten@users.noreply.github.com> Co-authored-by: athitten <athitten@users.noreply.github.com>

…#9726) * Option to convert response to OPenAI format Signed-off-by: Abhishree <abhishreetm@gmail.com> * Add OpenAI response arg and store_args_to_json method Signed-off-by: Abhishree <abhishreetm@gmail.com> * Apply isort and black reformatting Signed-off-by: athitten <athitten@users.noreply.github.com> --------- Signed-off-by: Abhishree <abhishreetm@gmail.com> Signed-off-by: athitten <athitten@users.noreply.github.com> Co-authored-by: athitten <athitten@users.noreply.github.com> Signed-off-by: Hainan Xu <hainanx@nvidia.com>

…#9726) * Option to convert response to OPenAI format Signed-off-by: Abhishree <abhishreetm@gmail.com> * Add OpenAI response arg and store_args_to_json method Signed-off-by: Abhishree <abhishreetm@gmail.com> * Apply isort and black reformatting Signed-off-by: athitten <athitten@users.noreply.github.com> --------- Signed-off-by: Abhishree <abhishreetm@gmail.com> Signed-off-by: athitten <athitten@users.noreply.github.com> Co-authored-by: athitten <athitten@users.noreply.github.com>

athitten added Run CICD and removed Run CICD labels Jul 14, 2024

athitten added 2 commits July 16, 2024 16:29

Option to convert response to OPenAI format

7846ce8

Signed-off-by: Abhishree <abhishreetm@gmail.com>

Add OpenAI response arg and store_args_to_json method

43db3ae

Signed-off-by: Abhishree <abhishreetm@gmail.com>

athitten force-pushed the athitten/openai_format_response branch from fcff757 to 43db3ae Compare July 16, 2024 23:29

Apply isort and black reformatting

53667bb

Signed-off-by: athitten <athitten@users.noreply.github.com>

athitten commented Jul 16, 2024

View reviewed changes

athitten marked this pull request as ready for review July 16, 2024 23:49

athitten added Run CICD and removed Run CICD labels Jul 16, 2024

athitten changed the title ~~Option to convert response to OpenAI format~~ Add option to convert PyTriton response to OpenAI format Jul 16, 2024

athitten requested a review from oyilmaz-nvidia July 17, 2024 00:00

oyilmaz-nvidia reviewed Jul 17, 2024

View reviewed changes

athitten changed the base branch from main to r2.0.0rc1 July 18, 2024 05:46

athitten changed the base branch from r2.0.0rc1 to main July 18, 2024 05:46

oyilmaz-nvidia approved these changes Jul 18, 2024

View reviewed changes

oyilmaz-nvidia merged commit 234ac8b into main Jul 18, 2024

oyilmaz-nvidia deleted the athitten/openai_format_response branch July 18, 2024 15:57

athitten mentioned this pull request Jul 18, 2024

Add OpenAI format response to r2.0.0rc1 #9796

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add option to convert PyTriton response to OpenAI format#9726

Add option to convert PyTriton response to OpenAI format#9726
oyilmaz-nvidia merged 3 commits intomainfrom
athitten/openai_format_response

athitten commented Jul 14, 2024 •

edited

Loading

Uh oh!

athitten Jul 16, 2024

Uh oh!

athitten Jul 16, 2024

Uh oh!

athitten Jul 16, 2024

Uh oh!

oyilmaz-nvidia Jul 17, 2024

Uh oh!

athitten Jul 17, 2024

Uh oh!

oyilmaz-nvidia Jul 18, 2024

Uh oh!

oyilmaz-nvidia left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Comments

Conversation

athitten commented Jul 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do ?

Changelog

Usage

GitHub Actions CI

Before your PR is "Ready for review"

Who can review?

Additional Information

Uh oh!

athitten Jul 16, 2024

Choose a reason for hiding this comment

Uh oh!

athitten Jul 16, 2024

Choose a reason for hiding this comment

Uh oh!

athitten Jul 16, 2024

Choose a reason for hiding this comment

Uh oh!

oyilmaz-nvidia Jul 17, 2024

Choose a reason for hiding this comment

Uh oh!

athitten Jul 17, 2024

Choose a reason for hiding this comment

Uh oh!

oyilmaz-nvidia Jul 18, 2024

Choose a reason for hiding this comment

Uh oh!

oyilmaz-nvidia left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Comments

athitten commented Jul 14, 2024 •

edited

Loading