Update fastspeech2 model card

[ricalanis]

#36979

• Updated the FastSpeech2Conformer model card
• Could not replicate the pipeline example from original code, as a bug related to FastSpeech2ConformerConfig appears (AttributeError: 'FastSpeech2ConformerConfig' object has no attribute 'model_config') still left it as is.
• Did not add terminal call out as I could not find a direct way to do that.
• Added a small section to use the combined model with the vocoder.
• Did not add quantization as the model architecture is not linear layer heavy (limited impact of quantization, AFAIK)
• This was a little bit more challenging, still. really thank you for your patience!

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• [] Did you write any new necessary tests? NA

[github-actions]

Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the Ready for review button (at the bottom of the PR page). This will assign reviewers and trigger CI.

[Rocketknight1]

cc @stevhliu

[stevhliu]

Thanks for your contribution! I've pinged our audio expert @eustlb to have a look at why the code examples are failing.

[stevhliu]

@eustlb, would you mind taking a look at why the code is failing here? It returns a AttributeError: 'FastSpeech2ConformerConfig' object has no attribute 'model_config' error which seems to be available here. Thanks!

# pip install -U -q g2p-en
import torch
import soundfile as sf
from transformers import pipeline, FastSpeech2ConformerHifiGan

vocoder = FastSpeech2ConformerHifiGan.from_pretrained("espnet/fastspeech2_conformer_hifigan")
synthesiser = pipeline(task="text-to-audio", model="espnet/fastspeech2_conformer", vocoder=vocoder, device=0, torch_dtype=torch.float16)
speech = synthesiser("Hello, my dog is cooler than you!")
sf.write("speech.wav", speech["audio"].squeeze(), samplerate=speech["sampling_rate"])

[eustlb]

Nice catch, let's simply modify modeling_audio.py's MODEL_FOR_TEXT_TO_WAVEFORM_MAPPING_NAMES

MODEL_FOR_TEXT_TO_WAVEFORM_MAPPING_NAMES = OrderedDict(
    [
        # Model for Text-To-Waveform mapping
        ("bark", "BarkModel"),
        ("csm", "CsmForConditionalGeneration"),
        ("fastspeech2_conformer", "FastSpeech2ConformerModel"),
        ("fastspeech2_conformer_with_hifigan", "FastSpeech2ConformerWithHifiGan"),
        ("musicgen", "MusicgenForConditionalGeneration"),
        ("musicgen_melody", "MusicgenMelodyForConditionalGeneration"),
        ("qwen2_5_omni", "Qwen2_5OmniForConditionalGeneration"),
        ("seamless_m4t", "SeamlessM4TForTextToSpeech"),
        ("seamless_m4t_v2", "SeamlessM4Tv2ForTextToSpeech"),
        ("vits", "VitsModel"),
    ]
)

[stevhliu]

Let's use the combined version so its easier:

# pip install -U -q g2p-en
import soundfile as sf
import torch
from transformers import AutoTokenizer, FastSpeech2ConformerWithHifiGan

tokenizer = AutoTokenizer.from_pretrained("espnet/fastspeech2_conformer")
model = FastSpeech2ConformerWithHifiGan.from_pretrained("espnet/fastspeech2_conformer_with_hifigan", torch_dtype=torch.float16, device_map="auto")

inputs = tokenizer("Hello, my dog is cute.", return_tensors="pt").to("cuda")
input_ids = inputs["input_ids"]

output_dict = model(input_ids, return_dict=True)
waveform = output_dict["waveform"]
sf.write("speech.wav", waveform.squeeze().detach().numpy(), samplerate=22050)

[ricalanis]

Done!