Improve: [newmm tokenizer] Change regular expression of "non-thai-characters"

[konbraphat51]

What does this changes

Make the newmm tokenization more accurate; recognize more characters as "non-thai"

What was wrong

#855
It sometimes didn't recognize non-thai symbols as non-thai
"(คนไม่เอา)" -> ['(คน', 'ไม่', 'เอา', ')']
"กม/ชม" -> ['กม', '/ชม']
"สีหน้า(รถ)" -> ['สีหน้า', '(รถ)']

How this fixes it

Fixed the recognition method of "non-thai-character".
The examples above are all improved.

Your checklist for this pull request

• Passed code styles and structures
• Passed code linting checks and unit test

[konbraphat51]

['👍👍👍 # ana', 'มาก', 'xxrep', 'xxwrep', 'น้อย', '.1146'] != ['xxwrep', '👍', '#', 'ana', 'มาก', 'xxrep', 'xxwrep', 'น้อย', '.', '1146']

It seems that this change makes the tokenization more minute than the test-case.
What do orginizers here think about this?

[wannaphong]

['👍👍👍 # ana', 'มาก', 'xxrep', 'xxwrep', 'น้อย', '.1146'] != ['xxwrep', '👍', '#', 'ana', 'มาก', 'xxrep', 'xxwrep', 'น้อย', '.', '1146']

It seems that this change makes the tokenization more minute than the test-case. What do orginizers here think about this?

Can you update the pull request? 9df5a4a

[konbraphat51]

Greetings PR check

Resource not accessible by integration

I don't understand this. It this error common in this project?

[konbraphat51]

['👍👍👍 # ana', 'มาก', 'xxrep', 'xxwrep', 'น้อย', '.1146'] != ['xxwrep', '👍', '#', 'ana', 'มาก', 'xxrep', 'xxwrep', 'น้อย', '.', '1146']

It seems that this change makes the tokenization more minute than the test-case. What do orginizers here think about this?

Can you update the pull request? 9df5a4a

Updated

[konbraphat51]

It seems that there is unit-test error occuring by 9df5a4a

Ignorable?

[wannaphong]

Ignorable

Yes, It's self-host issues but I don't have time to new setup. The unit-test by GitHub is look good https://github.com/PyThaiNLP/pythainlp/actions/runs/6718024461/job/18256943528

[coveralls]

coverage: 0.0% (-87.0%) from 86.983% when pulling a5b6db4 on konbraphat51:dev into 9df5a4a on PyThaiNLP:dev.

[wannaphong]

I add some rule to fixed the error. konbraphat51#2

[pep8speaks]

Hello @konbraphat51! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

• In the file pythainlp/tokenize/newmm.py:

Line 42:1: E122 continuation line missing indentation or outdented

Comment last updated at 2023-11-01 13:08:42 UTC

[konbraphat51]

I merged and modified @wannaphong PR. Please check.

[wannaphong]

OK. It look ['เวลา', ' ', '12:12pm ', 'มี', 'โปรโมชั่น', ' ', '11.11'] is error. ['เวลา', ' ', '12:12pm',' ', 'มี', 'โปรโมชั่น', ' ', '11.11'] is correct.

[wannaphong]

I fixed.

[konbraphat51]

In my case, fb3e7bb showed
['เวลา', ' ', '12', ':12pm ', 'มี', 'โปรโมชั่น', ' ', '11.11']

The last [^\u0E00-\u0E7F]+ scoups up all the non-thai character until it hits Thai character

[wannaphong]

In my case, fb3e7bb showed ['เวลา', ' ', '12', ':12pm ', 'มี', 'โปรโมชั่น', ' ', '11.11']

The last [^\u0E00-\u0E7F]+ scoups up all the non-thai character until it hits Thai character

3d889f7 showed

>>> from pythainlp.tokenize import word_tokenize
>>> word_tokenize("เวลา 12:12pm มีโปรโมชั่น 11.11")
['เวลา', ' ', '12:12pm', ' ', 'มี', 'โปรโมชั่น', ' ', '11.11']
>>> word_tokenize("กม/ชม")
['กม', '/', 'ชม']
>>> word_tokenize("(คนไม่เอา)")
['(', 'คน', 'ไม่', 'เอา', ')']
>>> word_tokenize("สีหน้า(รถ)")
['สีหน้า', '(', 'รถ', ')']

[konbraphat51]

In my case, fb3e7bb showed ['เวลา', ' ', '12', ':12pm ', 'มี', 'โปรโมชั่น', ' ', '11.11']
The last [^\u0E00-\u0E7F]+ scoups up all the non-thai character until it hits Thai character

3d889f7 showed

>>> from pythainlp.tokenize import word_tokenize
>>> word_tokenize("เวลา 12:12pm มีโปรโมชั่น 11.11")
['เวลา', ' ', '12:12pm', ' ', 'มี', 'โปรโมชั่น', ' ', '11.11']
>>> word_tokenize("กม/ชม")
['กม', '/', 'ชม']
>>> word_tokenize("(คนไม่เอา)")
['(', 'คน', 'ไม่', 'เอา', ')']
>>> word_tokenize("สีหน้า(รถ)")
['สีหน้า', '(', 'รถ', ')']

Oh sorry, I was testing by segment() function. Didn't notice there was a post-process.
3d889f7 This seems good to me :)

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
No Duplication information

[konbraphat51]

Added the commentation for further maintenance.

[wannaphong]

Awesome 💯

[bact]

Looks fine. Doesn't break the number grouping.