EN TN Fixes for Issue 166 (#207)

* Rebases the updated main

Signed-off-by: Simon Zuberek <[email protected]>

* Passes Pynini fails SP

Signed-off-by: Simon Zuberek <[email protected]>

* Adjustst the weights on the domain graph

Signed-off-by: Simon Zuberek <[email protected]>

* Enables semiotic classes for SP tests

Signed-off-by: Simon Zuberek <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Reweights the tokenizer

Signed-off-by: Simon Zuberek <[email protected]>

* Updates test cases

Signed-off-by: Simon Zuberek <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Updates test cases

Signed-off-by: Simon Zuberek <[email protected]>

* Cleans up ELECTRONIC tagger

Signed-off-by: Simon Zuberek <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Updates test cases

Signed-off-by: Simon Zuberek <[email protected]>

* Updates Jenkins

Signed-off-by: Simon Zuberek <[email protected]>

* Enables all CI tests

Signed-off-by: Simon Zuberek <[email protected]>

* Updates EN TN Cache

Signed-off-by: Simon Zuberek <[email protected]>

---------

Signed-off-by: Simon Zuberek <[email protected]>
Co-authored-by: Simon Zuberek <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

Author: zoobereq

Jenkinsfile

@@ -13,7 +13,7 @@ pipeline {
 
     AR_TN_CACHE='/home/jenkinsci/TestData/text_norm/ci/grammars/04-24-24-0'
     DE_TN_CACHE='/home/jenkinsci/TestData/text_norm/ci/grammars/06-03-24-0'
-    EN_TN_CACHE='/home/jenkinsci/TestData/text_norm/ci/grammars/08-02-24-0'
+    EN_TN_CACHE='/home/jenkinsci/TestData/text_norm/ci/grammars/08-15-24-0'
     ES_TN_CACHE='/home/jenkinsci/TestData/text_norm/ci/grammars/08-14-24-0'
     ES_EN_TN_CACHE='/home/jenkinsci/TestData/text_norm/ci/grammars/08-14-24-0'
     FR_TN_CACHE='/home/jenkinsci/TestData/text_norm/ci/grammars/06-04-24-0'

nemo_text_processing/text_normalization/en/data/electronic/domain.tsv

@@ -1,6 +1,7 @@
 .com	dot com
 .org	dot org
 .gov	dot gov
+.edu	dot edu
 .uk	dot UK
 .fr	dot FR
 .net	dot net

nemo_text_processing/text_normalization/en/taggers/electronic.py

@@ -1,4 +1,4 @@
-# Copyright (c) 2021, NVIDIA CORPORATION.  All rights reserved.
+# Copyright (c) 2024, NVIDIA CORPORATION.  All rights reserved.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -22,7 +22,6 @@
     NEMO_DIGIT,
     NEMO_NOT_SPACE,
     NEMO_SIGMA,
-    NEMO_SPACE,
     NEMO_UPPER,
     TO_UPPER,
     GraphFst,
@@ -55,6 +54,7 @@ def __init__(self, cardinal: GraphFst, deterministic: bool = True):
         cc_cues = pynutil.add_weight(pynini.string_file(get_abs_path("data/electronic/cc_cues.tsv")), MIN_NEG_WEIGHT)
 
         accepted_symbols = pynini.project(pynini.string_file(get_abs_path("data/electronic/symbol.tsv")), "input")
+
         accepted_common_domains = pynini.project(
             pynini.string_file(get_abs_path("data/electronic/domain.tsv")), "input"
         )
@@ -115,16 +115,19 @@ def __init__(self, cardinal: GraphFst, deterministic: bool = True):
             username + domain_graph_with_class_tags,
         )
 
-        # abc.com, abc.com/123-sm
-        # when only domain, make sure it starts and end with NEMO_ALPHA
-        graph |= (
+        full_stop_accep = pynini.accep(".")
+        dollar_accep = pynini.accep("$")  # Include for the correct transduction of the money graph
+        excluded_symbols = full_stop_accep | dollar_accep
+        filtered_symbols = pynini.difference(accepted_symbols, excluded_symbols)
+        accepted_characters = NEMO_ALPHA | NEMO_DIGIT | filtered_symbols
+        domain_component = full_stop_accep + pynini.closure(accepted_characters, 2)
+        graph_domain = (
             pynutil.insert('domain: "')
-            + pynini.compose(
-                NEMO_ALPHA + pynini.closure(NEMO_NOT_SPACE) + accepted_common_domains + pynini.closure(NEMO_NOT_SPACE),
-                domain_graph,
-            ).optimize()
+            + (pynini.closure(accepted_characters, 1) + pynini.closure(domain_component, 1))
             + pynutil.insert('"')
-        )
+        ).optimize()
+
+        graph |= pynutil.add_weight(graph_domain, MIN_NEG_WEIGHT)
 
         # www.abc.com/sdafsdf, or https://www.abc.com/asdfad or www.abc.abc/asdfad
         graph |= protocol + pynutil.insert(" ") + domain_graph_with_class_tags

nemo_text_processing/text_normalization/en/taggers/tokenize_and_classify.py

@@ -53,7 +53,7 @@ class ClassifyFst(GraphFst):
     Final class that composes all other classification grammars. This class can process an entire sentence including punctuation.
     For deployment, this grammar will be compiled and exported to OpenFst Finite State Archive (FAR) File.
     More details to deployment at NeMo/tools/text_processing_deployment.
-    
+
     Args:
         input_case: accepting either "lower_cased" or "cased" input.
         deterministic: if True will provide a single transduction option,
@@ -78,11 +78,11 @@ def __init__(
             os.makedirs(cache_dir, exist_ok=True)
             whitelist_file = os.path.basename(whitelist) if whitelist else ""
             far_file = os.path.join(
-                cache_dir, f"en_tn_{deterministic}_deterministic_{input_case}_{whitelist_file}_tokenize.far"
+                cache_dir, f"en_tn_{deterministic}_deterministic_{input_case}_{whitelist_file}_tokenize.far",
             )
         if not overwrite_cache and far_file and os.path.exists(far_file):
             self.fst = pynini.Far(far_file, mode="r")["tokenize_and_classify"]
-            logger.info(f'ClassifyFst.fst was restored from {far_file}.')
+            logger.info(f"ClassifyFst.fst was restored from {far_file}.")
         else:
             logger.info(f"Creating ClassifyFst grammars.")
 
@@ -107,7 +107,7 @@ def __init__(
             logger.debug(f"fraction: {time.time() - start_time: .2f}s -- {fraction_graph.num_states()} nodes")
 
             start_time = time.time()
-            measure = MeasureFst(cardinal=cardinal, decimal=decimal, fraction=fraction, deterministic=deterministic)
+            measure = MeasureFst(cardinal=cardinal, decimal=decimal, fraction=fraction, deterministic=deterministic,)
             measure_graph = measure.fst
             logger.debug(f"measure: {time.time() - start_time: .2f}s -- {measure_graph.num_states()} nodes")
 
@@ -157,7 +157,7 @@ def __init__(
             time_final = pynini.compose(time_graph, v_time_graph)
             date_final = pynini.compose(date_graph, v_date_graph)
             range_graph = RangeFst(
-                time=time_final, date=date_final, cardinal=cardinal, deterministic=deterministic
+                time=time_final, date=date_final, cardinal=cardinal, deterministic=deterministic,
             ).fst
             logger.debug(f"range: {time.time() - start_time: .2f}s -- {range_graph.num_states()} nodes")
 
@@ -171,10 +171,10 @@ def __init__(
                 | pynutil.add_weight(ordinal_graph, 1.1)
                 | pynutil.add_weight(money_graph, 1.1)
                 | pynutil.add_weight(telephone_graph, 1.1)
-                | pynutil.add_weight(electonic_graph, 1.1)
+                | pynutil.add_weight(electonic_graph, 1.11)
                 | pynutil.add_weight(fraction_graph, 1.1)
                 | pynutil.add_weight(range_graph, 1.1)
-                | pynutil.add_weight(serial_graph, 1.1003)  # should be higher than the rest of the classes
+                | pynutil.add_weight(serial_graph, 1.12)  # should be higher than the rest of the classes
             )
 
             # roman_graph = RomanFst(deterministic=deterministic).fst
@@ -187,7 +187,8 @@ def __init__(
             punct = pynutil.insert("tokens { ") + pynutil.add_weight(punct_graph, weight=2.1) + pynutil.insert(" }")
             punct = pynini.closure(
                 pynini.compose(pynini.closure(NEMO_WHITE_SPACE, 1), delete_extra_space)
-                | (pynutil.insert(" ") + punct),
+                | (pynutil.insert(" ") + punct)
+                | punct,
                 1,
             )
 

tests/nemo_text_processing/en/data_text_normalization/test_cases_electronic.txt

@@ -4,7 +4,7 @@ [email protected]~a at hotmail dot FR
 [email protected]~a at hotmail dot IT
 [email protected]~a at aol dot IT
 [email protected]~a at msn dot IT
-[email protected]~cdf at abc dot EDU
+[email protected]~cdf at abc dot edu
 [email protected]~abc at gmail dot ABC
 [email protected]~abc at abc dot com
 [email protected]~asdf one two three at abc dot com
@@ -13,9 +13,9 @@ [email protected]~ab three dot sdd dot three at gmail dot com
 email is [email protected]~email is abc one at gmail dot com
 [email protected]~abs at NVIDIA dot com
 email is a-1.b3_c&[email protected] test~email is a dash one dot b three underscore c ampersand dash d at gma four i dash l dot com test
-nvidia.com~NVIDIA dot com
+nvidia.com~nvidia dot com
 test.com~test dot com
-test.abc~test.abc
+test.abc~test dot abc
 http://www.ourdailynews.com.sm~HTTP colon slash slash WWW dot ourdailynews dot com dot SM
 https://www.ourdailynews.com.sm~HTTPS colon slash slash WWW dot ourdailynews dot com dot SM
 www.ourdailynews.com.sm~WWW dot ourdailynews dot com dot SM
@@ -24,8 +24,8 @@ [email protected]~sdf at gmail dot com dot SM
 [email protected]/123-sm~sdf at gmail dot com slash one two three dash SM
 [email protected]/123-sm~sdf at gmail dot ABC slash one two three dash SM
 [email protected]/123456-sm~sdf at gmail dot ABC slash one two three four five six dash SM
-ourdailynews.com/12-sm~ourdailynews dot com slash one two dash SM
-ourdailynews.abc~ourdailynews.abc
+ourdailynews.com/12-sm~ourdailynews dot com slash one two dash sm
+ourdailynews.abc~ourdailynews dot abc
 file:///c/code/NeMo/docs/build/html/processing/intro.html~file colon slash slash slash c slash code slash nemo slash docs slash build slash html slash processing slash intro dot HTML
 file:///photos/image.jpg~file colon slash slash slash photos slash image dot jpeg
 electronic test.com and test2.uk~electronic test dot com and test two dot UK
@@ -38,4 +38,5 @@ [email protected]~RTX pro help at exchange dot NVIDIA dot com
 [email protected]~enterprise services at NVIDIA dot com
 [email protected]~enterprise dash services at NVIDIA dot com
 https://www.nvidia.com/dgx-basepod/~HTTPS colon slash slash WWW dot NVIDIA dot com slash DGX dash BASEPOD slash
-i can use your card ending in 8876~i can use your card ending in eight eight seven six
+i can use your card ending in 8876~i can use your card ending in eight eight seven six
+here is mail.nasa.gov.~here is mail dot nasa dot gov.

tests/nemo_text_processing/en/data_text_normalization/test_cases_punctuation.txt

@@ -39,7 +39,6 @@ a: ``b``, ``c``, ``d``, ``e``, ``f`` or ``g``.~a: "b", "c", "d", "e", "f" or "g"
 "And now," he said, "He,~"And now," he said, "He,
 80's~eighty's
 test [email protected]~test. b at gmail dot com
-ourdailynews.abc/123-sm~ourdailynews. abc/one hundred twenty three-sm
 123,000012~one hundred and twenty three, zero zero zero zero one two
 9000,000th~nine thousand, zero zero zero th
 16 July, 1943~the sixteenth of july, nineteen forty three
@@ -59,6 +58,6 @@ dr. Evil~dr. Evil
 1!!!!~one!!!!
 (1)Hello~(one) Hello
 ÀÁÂÃ check §- and ƛ, also ɧ~ÀÁÂÃ check section - and ƛ, also ɧ
-Hi it's 5pm,4A.M.?-34. Hi,no,yes,34! 12,again,4 and NO?17 and $.01,here & there--0.004kg~Hi it's five PM, four AM.? minus thirty four. Hi,no,yes, thirty four! twelve, again, four and NO? seventeen and one cent, here and there - minus zero point zero zero four kilograms
+Hi it's 5pm,4A.M.?-34. Hi,no,yes,34! 12,again,4 and NO?17 and $.01,here & there -0.004kg~Hi it's five PM, four AM.? minus thirty four. Hi,no,yes, thirty four! twelve, again, four and NO? seventeen and one cent, here and there minus zero point zero zero four kilograms
 1°C.~one degree Celsius.
 my email is [email protected]!~my email is myemail at gmail dot com!

tests/nemo_text_processing/en/data_text_normalization/test_cases_punctuation_match_input.txt

@@ -10,4 +10,4 @@ a: ``b``,``c``, ``d``, ``e``, ``f`` or ``g``.~a: "b","c", "d", "e", "f" or "g".
 1 ,~one ,
 (1)Hello~(one)Hello
 ÀÁÂÃ check §- and ƛ, also ɧ~ÀÁÂÃ check section- and ƛ, also ɧ
-Hi it's 5pm,4A.M.?-34. Hi,no,yes,34! 12,again,4 and NO?17 and $.01,here & there--0.004kg~Hi it's five PM,four AM.?minus thirty four. Hi,no,yes,thirty four! twelve,again,four and NO?seventeen and one cent,here and there - minus zero point zero zero four kilograms
+Hi it's 5pm,4A.M.?-34. Hi,no,yes,34! 12,again,4 and NO?17 and $.01,here & there -0.004kg~Hi it's five PM,four AM.?minus thirty four. Hi,no,yes,thirty four! twelve,again,four and NO?seventeen and one cent,here and there minus zero point zero zero four kilograms