From 3cc808f9cb3672795d47c22c0cae5aa4f10f4ae1 Mon Sep 17 00:00:00 2001 From: Zhang Jun Date: Fri, 25 Jul 2025 16:04:11 +0800 Subject: [PATCH 1/3] update ernie model doc Signed-off-by: Zhang Jun --- docs/source/en/model_doc/ernie.md | 149 ++++++++++++++---------------- 1 file changed, 71 insertions(+), 78 deletions(-) diff --git a/docs/source/en/model_doc/ernie.md b/docs/source/en/model_doc/ernie.md index 596a7b1f4b38..2393914b471b 100644 --- a/docs/source/en/model_doc/ernie.md +++ b/docs/source/en/model_doc/ernie.md @@ -14,106 +14,99 @@ rendered properly in your Markdown viewer. --> -# ERNIE - -
-PyTorch +
+
+ PyTorch +
-## Overview -ERNIE is a series of powerful models proposed by baidu, especially in Chinese tasks, -including [ERNIE1.0](https://huggingface.co/papers/1904.09223), [ERNIE2.0](https://ojs.aaai.org/index.php/AAAI/article/view/6428), -[ERNIE3.0](https://huggingface.co/papers/2107.02137), [ERNIE-Gram](https://huggingface.co/papers/2010.12148), [ERNIE-health](https://huggingface.co/papers/2110.07244), etc. - -These models are contributed by [nghuyong](https://huggingface.co/nghuyong) and the official code can be found in [PaddleNLP](https://github.com/PaddlePaddle/PaddleNLP) (in PaddlePaddle). - -### Usage example -Take `ernie-1.0-base-zh` as an example: - -```Python -from transformers import AutoTokenizer, AutoModel -tokenizer = AutoTokenizer.from_pretrained("nghuyong/ernie-1.0-base-zh") -model = AutoModel.from_pretrained("nghuyong/ernie-1.0-base-zh") -``` - -### Model checkpoints - -| Model Name | Language | Description | -|:-------------------:|:--------:|:-------------------------------:| -| ernie-1.0-base-zh | Chinese | Layer:12, Heads:12, Hidden:768 | -| ernie-2.0-base-en | English | Layer:12, Heads:12, Hidden:768 | -| ernie-2.0-large-en | English | Layer:24, Heads:16, Hidden:1024 | -| ernie-3.0-base-zh | Chinese | Layer:12, Heads:12, Hidden:768 | -| ernie-3.0-medium-zh | Chinese | Layer:6, Heads:12, Hidden:768 | -| ernie-3.0-mini-zh | Chinese | Layer:6, Heads:12, Hidden:384 | -| ernie-3.0-micro-zh | Chinese | Layer:4, Heads:12, Hidden:384 | -| ernie-3.0-nano-zh | Chinese | Layer:4, Heads:12, Hidden:312 | -| ernie-health-zh | Chinese | Layer:12, Heads:12, Hidden:768 | -| ernie-gram-zh | Chinese | Layer:12, Heads:12, Hidden:768 | - -You can find all the supported models from huggingface's model hub: [huggingface.co/nghuyong](https://huggingface.co/nghuyong), and model details from paddle's official -repo: [PaddleNLP](https://paddlenlp.readthedocs.io/zh/latest/model_zoo/transformers/ERNIE/contents.html) -and [ERNIE](https://github.com/PaddlePaddle/ERNIE/blob/repro). - -## Resources +# ERNIE -- [Text classification task guide](../tasks/sequence_classification) -- [Token classification task guide](../tasks/token_classification) -- [Question answering task guide](../tasks/question_answering) -- [Causal language modeling task guide](../tasks/language_modeling) -- [Masked language modeling task guide](../tasks/masked_language_modeling) -- [Multiple choice task guide](../tasks/multiple_choice) +[ERNIE1.0](https://arxiv.org/abs/1904.09223), [ERNIE2.0](https://ojs.aaai.org/index.php/AAAI/article/view/6428), +[ERNIE3.0](https://arxiv.org/abs/2107.02137), [ERNIE-Gram](https://arxiv.org/abs/2010.12148), [ERNIE-health](https://arxiv.org/abs/2110.07244) are a series of powerful models proposed by baidu, especially in Chinese tasks. -## ErnieConfig +ERNIE (Enhanced Representation through kNowledge IntEgration) is designed to learn language representation enhanced by knowledge masking strategies, which includes entity-level masking and phrase-level masking. -[[autodoc]] ErnieConfig - - all +Other ERNIE models released by baidu can be found at [Ernie 4.5](./ernie4_5.md), and [Ernie 4.5 MoE](./ernie4_5_moe.md). -## Ernie specific outputs +> [!TIP] +> This model was contributed by [nghuyong](https://huggingface.co/nghuyong), and the official code can be found in [PaddleNLP](https://github.com/PaddlePaddle/PaddleNLP) (in PaddlePaddle). +> +> Click on the ERNIE models in the right sidebar for more examples of how to apply ERNIE to different language tasks. -[[autodoc]] models.ernie.modeling_ernie.ErnieForPreTrainingOutput +The example below demonstrates how to predict the `[MASK]` token with [`Pipeline`], [`AutoModel`], and from the command line. -## ErnieModel + + -[[autodoc]] ErnieModel - - forward +```py +from transformers import pipeline -## ErnieForPreTraining +pipeline = pipeline( + task="fill-mask", + model="nghuyong/ernie-3.0-xbase-zh" +) -[[autodoc]] ErnieForPreTraining - - forward +pipeline("巴黎是[MASK]国的首都。") +``` -## ErnieForCausalLM + + -[[autodoc]] ErnieForCausalLM - - forward +```py +import torch +from transformers import AutoModelForMaskedLM, AutoTokenizer -## ErnieForMaskedLM +tokenizer = AutoTokenizer.from_pretrained( + "nghuyong/ernie-3.0-xbase-zh", +) +model = AutoModelForMaskedLM.from_pretrained( + "nghuyong/ernie-3.0-xbase-zh", + torch_dtype=torch.float16, + device_map="auto" +) +inputs = tokenizer("巴黎是[MASK]国的首都。", return_tensors="pt").to("cuda") -[[autodoc]] ErnieForMaskedLM - - forward +with torch.no_grad(): + outputs = model(**inputs) + predictions = outputs.logits -## ErnieForNextSentencePrediction +masked_index = torch.where(inputs['input_ids'] == tokenizer.mask_token_id)[1] +predicted_token_id = predictions[0, masked_index].argmax(dim=-1) +predicted_token = tokenizer.decode(predicted_token_id) -[[autodoc]] ErnieForNextSentencePrediction - - forward +print(f"The predicted token is: {predicted_token}") +``` -## ErnieForSequenceClassification + + -[[autodoc]] ErnieForSequenceClassification - - forward +```bash +echo -e "巴黎是[MASK]国的首都。" | transformers run --task fill-mask --model nghuyong/ernie-3.0-xbase-zh --device 0 +``` -## ErnieForMultipleChoice + + -[[autodoc]] ErnieForMultipleChoice - - forward +## Notes -## ErnieForTokenClassification +Model variants are available in different sizes and languages. -[[autodoc]] ErnieForTokenClassification - - forward +| Model Name | Language | Description | +|:-------------------:|:--------:|:-------------------------------:| +| ernie-1.0-base-zh | Chinese | Layer:12, Heads:12, Hidden:768 | +| ernie-2.0-base-en | English | Layer:12, Heads:12, Hidden:768 | +| ernie-2.0-large-en | English | Layer:24, Heads:16, Hidden:1024 | +| ernie-3.0-base-zh | Chinese | Layer:12, Heads:12, Hidden:768 | +| ernie-3.0-medium-zh | Chinese | Layer:6, Heads:12, Hidden:768 | +| ernie-3.0-mini-zh | Chinese | Layer:6, Heads:12, Hidden:384 | +| ernie-3.0-micro-zh | Chinese | Layer:4, Heads:12, Hidden:384 | +| ernie-3.0-nano-zh | Chinese | Layer:4, Heads:12, Hidden:312 | +| ernie-health-zh | Chinese | Layer:12, Heads:12, Hidden:768 | +| ernie-gram-zh | Chinese | Layer:12, Heads:12, Hidden:768 | -## ErnieForQuestionAnswering +## Resources -[[autodoc]] ErnieForQuestionAnswering - - forward \ No newline at end of file +You can find all the supported models from huggingface's model hub: [huggingface.co/nghuyong](https://huggingface.co/nghuyong), and model details from paddle's official +repo: [PaddleNLP](https://paddlenlp.readthedocs.io/zh/latest/model_zoo/transformers/ERNIE/contents.html) +and [ERNIE's legacy branch](https://github.com/PaddlePaddle/ERNIE/tree/legacy/develop). From 6b2b1e6552c61521bbb5cd08a33a1663f3c1ece0 Mon Sep 17 00:00:00 2001 From: Zhang Jun Date: Fri, 25 Jul 2025 16:41:18 +0800 Subject: [PATCH 2/3] address ruff format error reported by ci Signed-off-by: Zhang Jun --- src/transformers/modeling_flash_attention_utils.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/src/transformers/modeling_flash_attention_utils.py b/src/transformers/modeling_flash_attention_utils.py index 0a91532cee88..47744eaca3f2 100644 --- a/src/transformers/modeling_flash_attention_utils.py +++ b/src/transformers/modeling_flash_attention_utils.py @@ -393,7 +393,9 @@ def _flash_attention_forward( query_states, key_states, value_states = fa_peft_integration_check( query_states, key_states, value_states, target_dtype ) - use_mask = position_ids is not None or all(k is not None for k in [cu_seq_lens_q, cu_seq_lens_k, max_length_q, max_length_k]) + use_mask = position_ids is not None or all( + k is not None for k in [cu_seq_lens_q, cu_seq_lens_k, max_length_q, max_length_k] + ) if attention_mask is not None: q, k, v, idx, (cu_q, cu_k), (mq, mk) = _upad_input( query_states, key_states, value_states, attention_mask, query_length, unpad_fn From 8a276a884977b35510c36c79bd1c085ba9f6755e Mon Sep 17 00:00:00 2001 From: Zhang Jun Date: Fri, 25 Jul 2025 16:42:02 +0800 Subject: [PATCH 3/3] address check_repository_consistency error reported by ci Signed-off-by: Zhang Jun --- docs/source/en/model_doc/ernie.md | 54 +++++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+) diff --git a/docs/source/en/model_doc/ernie.md b/docs/source/en/model_doc/ernie.md index 2393914b471b..8b2290ded685 100644 --- a/docs/source/en/model_doc/ernie.md +++ b/docs/source/en/model_doc/ernie.md @@ -110,3 +110,57 @@ Model variants are available in different sizes and languages. You can find all the supported models from huggingface's model hub: [huggingface.co/nghuyong](https://huggingface.co/nghuyong), and model details from paddle's official repo: [PaddleNLP](https://paddlenlp.readthedocs.io/zh/latest/model_zoo/transformers/ERNIE/contents.html) and [ERNIE's legacy branch](https://github.com/PaddlePaddle/ERNIE/tree/legacy/develop). + +## ErnieConfig + +[[autodoc]] ErnieConfig + - all + +## Ernie specific outputs + +[[autodoc]] models.ernie.modeling_ernie.ErnieForPreTrainingOutput + +## ErnieModel + +[[autodoc]] ErnieModel + - forward + +## ErnieForPreTraining + +[[autodoc]] ErnieForPreTraining + - forward + +## ErnieForCausalLM + +[[autodoc]] ErnieForCausalLM + - forward + +## ErnieForMaskedLM + +[[autodoc]] ErnieForMaskedLM + - forward + +## ErnieForNextSentencePrediction + +[[autodoc]] ErnieForNextSentencePrediction + - forward + +## ErnieForSequenceClassification + +[[autodoc]] ErnieForSequenceClassification + - forward + +## ErnieForMultipleChoice + +[[autodoc]] ErnieForMultipleChoice + - forward + +## ErnieForTokenClassification + +[[autodoc]] ErnieForTokenClassification + - forward + +## ErnieForQuestionAnswering + +[[autodoc]] ErnieForQuestionAnswering + - forward