Thoughts on automating the addition of canto_unique characters and phrases #4
Description
Because we don't have a reliable Cantonese tokenizer, all the phrases added to the classifier need to satisfy the following property, in addition to not being shared by Mandarin:
Frequency of the phrase is very low in an untokenized Mandarin corpus. We don't want incorrect word boudaries to mess up our classification. For example, if the phrases like 全家下棋 are common in Mandarin, we don't want to add the phrase 家下 to canto_unique. The classifier will incorrectly estimate the word boundary in Mandarin texts and judge a Mandarin text to be Cantonese.
A way to automate the additions
Words.hk provides lists of Cantonese phrases and characters as well as their common variants. It also has labels on whether the phrase is a 書面語. We first exclude any 書面語 words from our candidates for canto_unique. Then, we can search each of the remaining characters or phrases in a large Mandarin corpus (e.g. Wudao Corpus 2.0 publushed a 200GB corpus for the public). If the character/phrase appears under a certain frequency, we add it to the canto_unique list. Alternatively, we can generate such a list of candidates and manually review them before adding them to the list.
Shortcomings of the current classification model
Since we don't have a Cantonese tokenizer, we have to rely on a coarser statistical estimation. It might result in some false positives (classified as Cantonese but is actually Mandarin) and some false negatives (the Cantonese word is disguised under 假借 or it's simply not on our list). Under these constraints, the best approach is to increase the number of canto_unique words that are very rare in Mandarin. Another approach is to add more mando_unique words that appear very rarely in Cantonese. This would require us to lookup the words in a large Cantonese corpus (e.g. LIHKG forum). However, since the written Cantonese corpuses I'm aware of are generally much smaller than their Mandarin counterparts and represent a certain age-range, I think the former approach relying on Mandarin corpuses would yield a more reliable classification.