NeMo 2.0 In-framework deployment support#11233
Closed
oyilmaz-nvidia wants to merge 16 commits intomainfrom
Closed
Conversation
Signed-off-by: Hemil Desai <hemild@nvidia.com>
Signed-off-by: hemildesai <hemildesai@users.noreply.github.com>
Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>
…into onur/nemo2_inframework_support
Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>
Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>
Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>
Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>
Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>
Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>
Signed-off-by: oyilmaz-nvidia <oyilmaz-nvidia@users.noreply.github.com>
8 tasks
janekl
reviewed
Nov 19, 2024
|
|
||
| @staticmethod | ||
| def get_deployable( | ||
| nemo_checkpoint_filepath: str = None, |
Collaborator
There was a problem hiding this comment.
Typing should be Optional[str] here
janekl
reviewed
Nov 19, 2024
| nemo_checkpoint_filepath: str = None, | ||
| num_devices: int = 1, | ||
| num_nodes: int = 1, | ||
| tensor_model_parallel_size: int = 1, |
Collaborator
There was a problem hiding this comment.
Params num_devices & num_nodes are quite tied to tensor_model_parallel_size & pipeline_model_parallel_size.
How about making tensor_model_parallel_size & pipeline_model_parallel_size optional and setting them by default to num_devices & num_nodes, respectitvely (if they are None)?
Some assertion for consistency like num_devices * num_nodes == tensor_model_parallel_size * pipeline_model_parallel_size would also be useful.
(Not sure how context_parallel_size changes all this).
janekl
reviewed
Nov 19, 2024
|
|
||
|
|
||
| class MegatronLLMDeployableNemo2(ITritonDeployable): | ||
| """Triton inference server compatible deploy class for a .nemo model file""" |
Collaborator
There was a problem hiding this comment.
It is no longer a .nemo file I think, just a folder.
janekl
reviewed
Nov 19, 2024
| else: | ||
| output_log_probs.append(lp) | ||
| output_infer["log_probs"] = np.array(output_log_probs) | ||
| except Exception as error: |
Collaborator
There was a problem hiding this comment.
This possibly hides errors by inserting them in output directory. Can we just raise them right away, i.e. remove the try / catch clause?
janekl
reviewed
Nov 19, 2024
| if "sentences" in result_dict.keys(): | ||
| output = result_dict["sentences"] | ||
| else: | ||
| return "Unknown output keyword." |
Collaborator
|
Left some minor comments, overall LGTM |
Contributor
|
This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days. |
Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com>
…VIDIA/NeMo into onur/nemo2_inframework_support
Contributor
|
beep boop 🤖: 🙏 The following files have warnings. In case you are familiar with these, please try helping us to improve the code base. Your code was analyzed with PyLint. The following annotations have been identified: Thank you for improving NeMo's documentation! |
Collaborator
Author
|
Closing this since there is a newer version of this PR: #11523 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do ?
Adds inframework deployment support and log probability output from trt-llm