Add Vietnamese measure text normalization support

[folivoramanh]

• Added measure tagger and verbalizer for Vietnamese TN
• Updated money tagger and verbalizer to handle per-unit measurements
• Added test cases for measure normalization
• Updated fraction handling for better integration
• Added data files for measurements, prefixes, and per-unit bases

What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

Before your PR is "Ready for review"

Pre checks:

• Have you signed your commits? Use git commit -s to sign.
• Do all unittests finish successfully before sending PR?
  1. pytest or (if your machine does not have GPU) pytest --cpu from the root folder (given you marked your test cases accordingly @pytest.mark.run_only_on('CPU')).
  2. Sparrowhawk tests bash tools/text_processing_deployment/export_grammars.sh --MODE=test ...
• If you are adding a new feature: Have you added test cases for both pytest and Sparrowhawk here.
• Have you added __init__.py for every folder and subfolder, including data folder which has .TSV files?
• Have you followed codeQL results and removed unused variables and imports (report is at the bottom of the PR in github review box) ?
• Have you added the correct license header Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. to all newly added Python files?
• If you copied nemo_text_processing/text_normalization/en/graph_utils.py your header's second line should be Copyright 2015 and onwards Google, Inc.. See an example here.
• Remove import guards (try import: ... except: ...) if not already done.
• If you added a new language or a new feature please update the NeMo documentation (lives in different repo).
• Have you added your language support to tools/text_processing_deployment/pynini_export.py.

PR Type:

• New Feature
• Bugfix
• Documentation
• Test

If you haven't finished some of the above items you can still open "Draft" PR.

[tbartley94]

Any way you can insert preserve_order in any of these or too complicated?

[folivoramanh]

not applicable for cardinal number patterns in this context. Cardinal numbers have deterministic structure: Vietnamese cardinal numbers follow a fixed left-to-right magnitude ordering (millions → thousands → hundreds → units). There's no alternative ordering that needs preservation.

[tbartley94]

if it's deterministic, then you can enforce order preservation no? this will make the graph more efficient.

[folivoramanh]

preserve_order isn't applicable here. Vietnamese cardinal numbers only output a single integer property (plus optional negative). Since cardinals have deterministic single-property output, there's nothing to preserve - no permutation occurs.

[tbartley94]

why are we using alias imports?

[folivoramanh]

The alias imports are being used because there are name conflicts between taggers and verbalizers. And those verbalizers are imported to build range_fst

[tbartley94]

i'm confused how this significantly differs from current postprocessing behavior.

[folivoramanh]

Here's example in English when have not post processing

NeMo-text-processing :: DEBUG    :: tokens { name: "It's" } tokens { electronic { domain: "123.000USD" } } tokens { name: "(" } tokens { measure { cardinal { integer: "one" } units: "kilogram" } } tokens { name: ")." } tokens { name: "kinda" } tokens { name: "good," } tokens { name: "(" } tokens { cardinal { integer: "one hundred and twenty three" } } tokens { name: ")" } tokens { name: "thanks." }
It's one two three dot zero zero zero USD ( one kilogram ). kinda good, ( one hundred and twenty three ) thanks.

when punct near other class not word, system recognized it as seperated tokens and when verbalized, output += ' ' + Normalizer.select_verbalizer(verbalizer_lattice) (in normalize.py) add space in every punct and other class (such as "(" and one kilogram). Thus post_processing.py fixes spacing issues inherent to the token-by-token verbalization architecture

[tbartley94]

LGTM

[tbartley94]

LGTM