Conversation
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
for more information, see https://pre-commit.ci
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
for more information, see https://pre-commit.ci
yaoyu-33
changed the title
|
jenkins |
|
jenkins |
1 similar comment
|
jenkins |
| restore_from_path: null # used when starting from a .nemo file | ||
|
|
||
| trainer: | ||
| devices: 2 |
There was a problem hiding this comment.
Why make changes to this file ?
There was a problem hiding this comment.
default conversion script uses this yaml file and use 1 gpu by default. and other models like https://github.com/NVIDIA/NeMo/blob/36a31c03497985c8f2165d4f9be788b574443391/examples/nlp/language_modeling/conf/megatron_gpt_config.yaml uses 1 gpu as well. Is there a reason to do 2?
| # - /raid/data/pile/my-gpt3_00_text_document | ||
| # - .5 | ||
| # - /raid/data/pile/my-gpt3_01_text_document | ||
| data_prefix: ??? |
There was a problem hiding this comment.
similar to above reason, try to make conversion work easier, since it's directly load this yaml. If put ??? it will raise an error while loading I think
| ffn_hidden_size: 3072 # Transformer FFN hidden size. Usually 4 * hidden_size. | ||
| num_attention_heads: 12 | ||
| skip_head: False | ||
| transformer_block_type: post_ln |
There was a problem hiding this comment.
previous default is preln. you want me to change it back? I am okay with it
| activations_checkpoint_layers_per_pipeline=None, | ||
| layernorm_epsilon=1e-5, | ||
| normalization='layernorm', | ||
| transformer_block_type='pre_ln', |
There was a problem hiding this comment.
Not necessary to change, but should we consider making these values enums ? So that people dont do typos ?
There was a problem hiding this comment.
yeah, that will affect all models, maybe add in nemo refactor plan
|
|
||
| if skip_head: | ||
| self.post_process = False | ||
| if self.post_process: |
There was a problem hiding this comment.
if not skip_head and if self.post_process : ?? Or maybe if self.post_process is used in other areas just leave it as such.
There was a problem hiding this comment.
yeah, it affect other places as well. But I can change it all to if not skip_head and if self.post_process. Do you want me to change? I remember current way is cleaner in terms of code but not necessarily more readable
There was a problem hiding this comment.
Well, if you think its cleaner then leave it as such.
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
|
jenkins |
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
|
jenkins |
4 participants
What does this PR do ?
Add BERT HF checkpoint converter and HF Bert Support in NeMo
Collection: [Note which collection this PR will affect]
Changelog
Usage
# Add a code snippet demonstrating how to use thisJenkins CI
To run Jenkins, a NeMo User with write access must comment
jenkinson the PR.Before your PR is "Ready for review"
Pre checks:
PR Type:
If you haven't finished some of the above items you can still open "Draft" PR.
Who can review?
Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.
Additional Information