DE TN Fixes #177

zoobereq · 2024-05-29T22:00:59Z

What does this PR do ?

This PR implements DE TN fixes for the following issues:

Adds support for normalizing social media tags (e.g. @zoobereq and @zoobereq.net)
Adds support for normalizing comma-separated digit strings
Adds support for period-separated time formats (e.g. 2.30 and 02.30)
Fixes the issue where the final period in sentences ending with a domain name would normalize as part of that domain.

This PR does not address the following:

The issue with optionally period-separated cardinals (e.g. 1 Mil = 100000 | 1.000.000) , which the currently implemented DE TN/ITN doesn't support. The TN and ITN taggers have been rebuilt to accommodate for that hiccup, but since cardinals plug into almost all other DE classes, this issue will be addressed gradually as other classes are fixed/(re)developed.

Before your PR is "Ready for review"

Pre checks:

PR Type:

New Feature
Bugfix
Documentation
Test

If you haven't finished some of the above items you can still open "Draft" PR.

Signed-off-by: Simon Zuberek <szuberek@nvidia.com>

…git strings Signed-off-by: Simon Zuberek <szuberek@nvidia.com>

…00.00 Uhr or 0.00 Uhr Signed-off-by: Simon Zuberek <szuberek@nvidia.com>

…ng with a domain name would be tagged as part of that domain name Signed-off-by: Simon Zuberek <szuberek@nvidia.com>

for more information, see https://pre-commit.ci

nemo_text_processing/text_normalization/de/taggers/electronic.py

Signed-off-by: Simon Zuberek <szuberek@nvidia.com>

…cessing into DE_TN_Fixes

for more information, see https://pre-commit.ci

Signed-off-by: Simon Zuberek <szuberek@nvidia.com>

for more information, see https://pre-commit.ci

ekmb · 2024-05-30T21:25:43Z

@zoobereq please update grammars path in Jenkins to re-built CI grammars https://github.com/NVIDIA/NeMo-text-processing/blob/main/Jenkinsfile#L15

Signed-off-by: Simon Zuberek <szuberek@nvidia.com>

for more information, see https://pre-commit.ci

Signed-off-by: Simon Zuberek <szuberek@nvidia.com>

nemo_text_processing/text_normalization/de/taggers/electronic.py

nemo_text_processing/text_normalization/de/taggers/time.py

nemo_text_processing/text_normalization/de/taggers/tokenize_and_classify.py

nemo_text_processing/text_normalization/de/verbalizers/electronic.py

bonham79 · 2024-06-03T22:11:21Z

tests/nemo_text_processing/de/data_text_normalization/test_cases_electronic.txt

+w w w punkt a m a z o n punkt com punkt de .~www.amazon.com.de.
+h t t p s doppelpunkt slash slash w w w punkt a b c punkt com slash a b fragezeichen gleichheitszeichen drei bindestrich slash a b s slash eins~https://www.abc.com/ab?=3-/abs/1
+at z u c k~@zuck
+at z o o b e r e q~@zoobereq


Don't use your name as an example. It leads to potential doxing.

Good point - thank you. Fixed.

tests/nemo_text_processing/de/test_sparrowhawk_inverse_text_normalization.sh

tests/nemo_text_processing/de/test_sparrowhawk_normalization.sh

Signed-off-by: Simon Zuberek <szuberek@nvidia.com>

tbartley94

Remove references to yourself and competitors and LGTM

tests/nemo_text_processing/de/data_text_normalization/test_cases_electronic.txt

Signed-off-by: Simon Zuberek <szuberek@nvidia.com>

tbartley94 · 2024-06-06T17:46:17Z

@zoobereq LGTM, will approve once Evelina is happy

ekmb

Thanks!

@zoobereq

* Adds support for social media tags (e.g. @zoobereq) Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * Adds test cases for social media tags Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * Fixes pathing for Sparrowhawk Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * Fixes the issue of the DE normalizer not accepting comma-separated digit strings Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * Fixes the issue where the normalizer didn't accept time formatted as 00.00 Uhr or 0.00 Uhr Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * Fixes the issue where the the sentence-final period in sentences ending with a domain name would be tagged as part of that domain name Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Removes unused imports Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fixes the formatting Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fixes #166 for DE Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Updates grammar paths Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * Minor Fixes Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * Fixes test cases Signed-off-by: Simon Zuberek <szuberek@nvidia.com> --------- Signed-off-by: Simon Zuberek <szuberek@nvidia.com> Co-authored-by: Simon Zuberek <szuberek@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: Alex Cui <alexcui1994@gmail.com>

@zoobereq

* Adds support for social media tags (e.g. @zoobereq) Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * Adds test cases for social media tags Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * Fixes pathing for Sparrowhawk Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * Fixes the issue of the DE normalizer not accepting comma-separated digit strings Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * Fixes the issue where the normalizer didn't accept time formatted as 00.00 Uhr or 0.00 Uhr Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * Fixes the issue where the the sentence-final period in sentences ending with a domain name would be tagged as part of that domain name Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Removes unused imports Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fixes the formatting Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fixes #166 for DE Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Updates grammar paths Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * Minor Fixes Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * Fixes test cases Signed-off-by: Simon Zuberek <szuberek@nvidia.com> --------- Signed-off-by: Simon Zuberek <szuberek@nvidia.com> Co-authored-by: Simon Zuberek <szuberek@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: Alex Cui <alexcui1994@gmail.com>

@zoobereq

* Adds support for social media tags (e.g. @zoobereq) Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * Adds test cases for social media tags Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * Fixes pathing for Sparrowhawk Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * Fixes the issue of the DE normalizer not accepting comma-separated digit strings Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * Fixes the issue where the normalizer didn't accept time formatted as 00.00 Uhr or 0.00 Uhr Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * Fixes the issue where the the sentence-final period in sentences ending with a domain name would be tagged as part of that domain name Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Removes unused imports Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fixes the formatting Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fixes #166 for DE Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Updates grammar paths Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * Minor Fixes Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * Fixes test cases Signed-off-by: Simon Zuberek <szuberek@nvidia.com> --------- Signed-off-by: Simon Zuberek <szuberek@nvidia.com> Co-authored-by: Simon Zuberek <szuberek@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: Alex Cui <alexcui1994@gmail.com>

@zoobereq

* Adds support for social media tags (e.g. @zoobereq) Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * Adds test cases for social media tags Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * Fixes pathing for Sparrowhawk Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * Fixes the issue of the DE normalizer not accepting comma-separated digit strings Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * Fixes the issue where the normalizer didn't accept time formatted as 00.00 Uhr or 0.00 Uhr Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * Fixes the issue where the the sentence-final period in sentences ending with a domain name would be tagged as part of that domain name Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Removes unused imports Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fixes the formatting Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fixes #166 for DE Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Updates grammar paths Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * Minor Fixes Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * Fixes test cases Signed-off-by: Simon Zuberek <szuberek@nvidia.com> --------- Signed-off-by: Simon Zuberek <szuberek@nvidia.com> Co-authored-by: Simon Zuberek <szuberek@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: Alex Cui <alexcui1994@gmail.com>

@zoobereq

* Adds support for social media tags (e.g. @zoobereq) Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * Adds test cases for social media tags Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * Fixes pathing for Sparrowhawk Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * Fixes the issue of the DE normalizer not accepting comma-separated digit strings Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * Fixes the issue where the normalizer didn't accept time formatted as 00.00 Uhr or 0.00 Uhr Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * Fixes the issue where the the sentence-final period in sentences ending with a domain name would be tagged as part of that domain name Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Removes unused imports Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fixes the formatting Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fixes #166 for DE Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Updates grammar paths Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * Minor Fixes Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * Fixes test cases Signed-off-by: Simon Zuberek <szuberek@nvidia.com> --------- Signed-off-by: Simon Zuberek <szuberek@nvidia.com> Co-authored-by: Simon Zuberek <szuberek@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: Alex Cui <alexcui1994@gmail.com>

@zoobereq

* Adds support for social media tags (e.g. @zoobereq) Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * Adds test cases for social media tags Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * Fixes pathing for Sparrowhawk Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * Fixes the issue of the DE normalizer not accepting comma-separated digit strings Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * Fixes the issue where the normalizer didn't accept time formatted as 00.00 Uhr or 0.00 Uhr Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * Fixes the issue where the the sentence-final period in sentences ending with a domain name would be tagged as part of that domain name Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Removes unused imports Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fixes the formatting Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fixes NVIDIA#166 for DE Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Updates grammar paths Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * Minor Fixes Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * Fixes test cases Signed-off-by: Simon Zuberek <szuberek@nvidia.com> --------- Signed-off-by: Simon Zuberek <szuberek@nvidia.com> Co-authored-by: Simon Zuberek <szuberek@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: Namrata Gachchi <ngachchi@nvidia.com>

@zoobereq

* Adds support for social media tags (e.g. @zoobereq) Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * Adds test cases for social media tags Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * Fixes pathing for Sparrowhawk Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * Fixes the issue of the DE normalizer not accepting comma-separated digit strings Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * Fixes the issue where the normalizer didn't accept time formatted as 00.00 Uhr or 0.00 Uhr Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * Fixes the issue where the the sentence-final period in sentences ending with a domain name would be tagged as part of that domain name Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Removes unused imports Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fixes the formatting Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fixes NVIDIA#166 for DE Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Updates grammar paths Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * Minor Fixes Signed-off-by: Simon Zuberek <szuberek@nvidia.com> * Fixes test cases Signed-off-by: Simon Zuberek <szuberek@nvidia.com> --------- Signed-off-by: Simon Zuberek <szuberek@nvidia.com> Co-authored-by: Simon Zuberek <szuberek@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

zoobereq and others added 7 commits May 29, 2024 14:55

Adds support for social media tags (e.g. @zoobereq)

8f05a5d

Signed-off-by: Simon Zuberek <szuberek@nvidia.com>

Adds test cases for social media tags

91bd0dc

Signed-off-by: Simon Zuberek <szuberek@nvidia.com>

Fixes pathing for Sparrowhawk

2433445

Signed-off-by: Simon Zuberek <szuberek@nvidia.com>

Fixes the issue of the DE normalizer not accepting comma-separated di…

34ea900

…git strings Signed-off-by: Simon Zuberek <szuberek@nvidia.com>

Fixes the issue where the normalizer didn't accept time formatted as …

a961941

…00.00 Uhr or 0.00 Uhr Signed-off-by: Simon Zuberek <szuberek@nvidia.com>

Fixes the issue where the the sentence-final period in sentences endi…

28a90ae

…ng with a domain name would be tagged as part of that domain name Signed-off-by: Simon Zuberek <szuberek@nvidia.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

35d9e6f

for more information, see https://pre-commit.ci

github-advanced-security bot found potential problems May 29, 2024

View reviewed changes

nemo_text_processing/text_normalization/de/taggers/electronic.py Fixed Show fixed Hide fixed

zoobereq and others added 5 commits May 30, 2024 08:33

Removes unused imports

39dd477

Signed-off-by: Simon Zuberek <szuberek@nvidia.com>

Merge branch 'DE_TN_Fixes' of https://github.com/NVIDIA/NeMo-text-pro…

1d54621

…cessing into DE_TN_Fixes

[pre-commit.ci] auto fixes from pre-commit.com hooks

8fe2eff

for more information, see https://pre-commit.ci

Fixes the formatting

7284b5a

Signed-off-by: Simon Zuberek <szuberek@nvidia.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

31d5824

for more information, see https://pre-commit.ci

zoobereq requested review from ekmb and tbartley94 May 30, 2024 19:04

zoobereq self-assigned this May 30, 2024

zoobereq marked this pull request as ready for review May 30, 2024 19:05

zoobereq and others added 3 commits June 3, 2024 09:58

Fixes #166 for DE

f24e685

Signed-off-by: Simon Zuberek <szuberek@nvidia.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

8fbc6f3

for more information, see https://pre-commit.ci

Updates grammar paths

71171dd

Signed-off-by: Simon Zuberek <szuberek@nvidia.com>

bonham79 suggested changes Jun 3, 2024

View reviewed changes

zoobereq changed the title ~~DE_TN_Fixes~~ DE TN Fixes Jun 4, 2024

Minor Fixes

4360b8e

Signed-off-by: Simon Zuberek <szuberek@nvidia.com>

tbartley94 previously approved these changes Jun 4, 2024

View reviewed changes

tests/nemo_text_processing/de/data_text_normalization/test_cases_electronic.txt Outdated Show resolved Hide resolved

tests/nemo_text_processing/de/data_text_normalization/test_cases_electronic.txt Outdated Show resolved Hide resolved

Fixes test cases

d2fe107

Signed-off-by: Simon Zuberek <szuberek@nvidia.com>

zoobereq dismissed tbartley94’s stale review via d2fe107 June 5, 2024 13:35

ekmb approved these changes Jun 6, 2024

View reviewed changes

ekmb merged commit e2fbc45 into main Jun 6, 2024

ekmb deleted the DE_TN_Fixes branch June 10, 2024 19:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DE TN Fixes #177

DE TN Fixes #177

Uh oh!

zoobereq commented May 29, 2024 •

edited

Loading

Uh oh!

Uh oh!

ekmb commented May 30, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bonham79 Jun 3, 2024

Uh oh!

zoobereq Jun 3, 2024 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

tbartley94 left a comment

Uh oh!

Uh oh!

Uh oh!

tbartley94 commented Jun 6, 2024

Uh oh!

ekmb left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

DE TN Fixes #177

DE TN Fixes #177

Uh oh!

Conversation

zoobereq commented May 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do ?

Before your PR is "Ready for review"

Uh oh!

Uh oh!

ekmb commented May 30, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bonham79 Jun 3, 2024

Choose a reason for hiding this comment

Uh oh!

zoobereq Jun 3, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

tbartley94 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

tbartley94 commented Jun 6, 2024

Uh oh!

ekmb left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

zoobereq commented May 29, 2024 •

edited

Loading

zoobereq Jun 3, 2024 •

edited

Loading