Capitalization is a kind-of pseduo-morphology

[linas]

This is a meta-issue, design-change request, to treat capitalization (and possibly other things) as a kind-of pseudo-morphology. See issue #42 for context. The general issue is about refomulating tokenization (and related issues) into a collection of rules (that could be encoded in a file). The example is capitalization.

For example: capitalization: we have a rule (coded in C) that if a word is at the beginning of a sentence, then we should search for a lower-case version of it. ... or if the word is after a semicolon, then we should search for a lower-case version of it. ... or if the word is after a quote, then we should search for a lower-case version of it. (Lincoln said, "Four-score and seven...") There is an obvious solution: design a new file, and place semi-colons, quotes and LEFT-WORD as markers for capitalization. Maybe this is kind-of-like a "new kind of affix rule" ?? All of the other affix rules state "if there is a certain sequence, then insert whitespace" while this new rule is "if there is a certain sequence, then look for downcase".

In the language learning code, I don't downcase any data in advance. Instead, the system eventually learns that certain words behave the same way, grammatically, whether they are uppercased or not. The system is blind to uppercasing: it just sees two different UTF8 strings that happen to fall into the same grammatical class.

To "solve" this problem, one can imagine three steps. First, a "morphological" analysis: given a certain grammatical class, compare pairs to strings to see if they have a common substring - for example, if the whole string matches, except for the first letter. This would imply that some words have a "morphology", where the first letter can be either one of two, while the rest of the word is the same.

The second step is to realize that there is a meta-morphology-rule, which states that there are many words, all of which have the property that they can begin with either one of two different initial letters. The correct choice of the initial letter depends on whether the preceding token was a semicolon, a quote, or the left-wall.

The third step is to realize that the meta-morphology-rule can be factored into approximately 26 different classes. That is, in principle, there are 52-squared/2=1352 possible sets containing two (initial) letters. Of these, only 26 are seen: {A, a}, {B, b}, {C, c} ....and one never ever sees {P, Q} or {Z, a}.

As long was we write C code, and know in advance that we are dealing with capital letters, then we can use pre-defined POSIX locales for capitalization. I'm trying to take two or three steps backwards, here. One is to treat capitalization as a kind of morphology, just like any other kind of morphology. The second is to create morphology classes - the pseudo-morpheme A is only substitutable by the pseudo-morpheme a. The third is that all of this should be somehow rule-driven and "generic" in some way.

The meta-meta-meta issue is that I want to expand the framework beyond just language written as UTF8 strings, but more generally, language with associated intonation, affect, facial expressions, or "language" from other domains (biology, spatial relations, etc.)

[ampli]

I once posted a proposal to handle capitalization as a special kind of morpheme:
https://groups.google.com/d/msg/link-grammar/hmK5gjXYWbk/6FC_x_43xpMJ

Note that the current tokenizer already implements that (untested, as the dict part of this idea has never been implemented).

This seems to me simple and natural way to do that, still using a pure link-grammar technic.

In the language learning code, I don't downcase any data in advance. Instead, the system eventually learns that certain words behave the same way, grammatically, whether they are uppercased or not. The system is blind to uppercasing: it just sees two different UTF8 strings that happen to fall into the same grammatical class.

The system find these words are very similar, but with a feature that is different. I propose to encode this different feature as a morpheme. All the capitalization rules will then can inferred (in principle) automatically as link-grammar rules (and until they are inferred automatically, they can be straightforwardly hand coded).

compare pairs to strings to see if they have a common substring

What I essentially propose is to look on substring of words as only a special case of morphemes.
The general case is that every feature of a word may be encoded as morpheme (I.e. an entity carrying disjuncts). It seems you agree with me, as you mention voice features of a word.
So I have never understood why you seemed to reject my proposal.

BTW, in Hebrew morphemes are not just substring of words. A morphemes that are "templates" are common (and they are really called "morphemes"). So I didn't invent something too new in my proposal to refer to capitalization (and vocalness of start of words, for a/an) as morphemes.

[ampli]

In the language learning code, I don't downcase any data in advance. Instead, the system eventually learns that certain words behave the same way, grammatically, whether they are uppercased or not. The system is blind to uppercasing: it just sees two different UTF8 strings that happen to fall into the same grammatical class.

The letter case is a private case of a more general problem.
For example, in Hebrew some letters change their look when they found at the end of words, and hence have a different code point even though essentially they are the "same letter" in a morphism standpoint.

[ampli]

I'm finally trying to implement hanfling of capitalized words by the dict.
I encountered a problem: How to generally implement a downcasing rule.
In English it is simple, and is hard-coded using tolower().
However, on French and maybe (I don't know) other languages it may be more complex.

Possible solution:
A hard coded tolower(), than a conversion according to affix-file definitions in order to generate alternatives.
E.g.:
E é è ê ë: TOLOWER+;

[ampli]

I implemented the main part of the pseudo-morphology capitalization.
There were several options for how and were to make the driving definition, and I chose to put it in 4.0.regex and a way that can be extended to phology or other uses (of course that everything is subject to drastic changes...):

% Treat a word match as morpheme-like feature, by prepending a token (the
% regex name) before a matching word.
%
% The following regex is used for prepending a "capitalization mark" before
% the matching word.
% The 'f' flag signifies prepending the regex name as a token before the word.
% The 'c' flag signifies to always continue to match the rest of the regexes.
% The 'l' flag signifies that the first letter should be converted to lowercase
% using the TOLOWER affix class, with a tolower() fallback.
<CAP-MARK>:     /^[[:upper:]][^'’]*[^[:punct:]]$/fcl

% The rest of this file contains regular expressions that are used to match
% tokens not found in the dictionary. Each regex is given a name which
% ...

I also made the needed changes, mainly in tokenize,.c, regex-morph.c and read-regex.c.

The easiest part was adding support for that in the dict.
The new code doesn't check at all for common-entities - instead, this is supported by the dict:

<marker-common-entity>: XXXGIVEN- & (<CAPITALIZED-WORDS> or <PL-CAPITALIZED-WORDS>);
<CAP-MARK>: CW- or XXXGIVEN+;
LEFT-WALL:
  {CW+} & (...);
<colon>: {CW+} & (...);
% etc.

Since the capitalized word is downcased if it is in the dict, it appears downcased even if it is a common-entity. It means that the code will need to handle restoring capitalization (like the old code).

However, the real (very) hard work has still not done:

1. Sane-morphism - if the feature-mark (<CAP-MARK> in that case) and/or the word after it are null, they are together to be considered as a single null word. I still don't know how to do that. It should work also in island_ok=1.
2. do_count() null counting: The <CAP-MARK> and the word after it are to be considered one unit for null-count (else the null count may be bogus).
3. After sane-morphism, there is a need to remove the feature-marks and optionally modify the display for null words.
   For example, if <CAP-MARK> is a null word but the word after it has a linkage, the word display should be converted to [word] and its links omitted.

Since these 3 tasks are really hard to implement, I first thought to combine the disjuncts of the feature token ( <CAP-MARK>) with those of the word after it. This way there is no need for tasks 1-3 above.
However, the word may then need 2 links to its preceding word (for example both Wd- and CW- to LEFT-WALL), which is currently not supported (it may be interesting to add such support).

Current linkage result examples:

                 +------------------QUc------------------+
                 +----------->WV----------->+            |
    +---->WV---->+-------->Wd---------+     +---Ost--+   |
    +->Wd--+--Ss-+-QUd+--CW--+        +-Ss*b+  +Ds**c+   |
    |      |     |    |      |        |     |  |     |   |
LEFT-WALL he said.q-d " <CAP-MARK> this.p is.v a  test.n "


    +------------------Xp------------------+----------------Xp----------------+
    +----------->WV---------->+            +--------->WV-------->+            |
    +-------->Wd--------+     +---Ost--+   +------>Wd------+     +---Osm--+   |
    +----CW----+        +-Ss*b+  +Ds**c+   +--CW--+        +-Ss*b+  +Ds**c+   |
    |          |        |     |  |     |   |      |        |     |  |     |   |
LEFT-WALL <CAP-MARK> this.p is.v a  test.n . <CAP-MARK> this.p is.v a  test.n .


    +--------->WV-------->+
    +------->Wd-------+   +------PP-----+--------MVp--------+----------Js----------+
    +----CW----+      +-Ss+     +---E---+-MVp-+--Js--+      |        +---XXXGIVEN--+
    |          |      |   |     |       |     |      |      |        |             |
LEFT-WALL <CAP-MARK> he 's.v usually gone.v to.r Boston.m for.p <CAP-MARK> thanksgiving.n-u


    +--------->WV--------->+                                      +---------------Bs--------------+
    +------->Wd-------+    +---------IV------->+                  |                +------RS------+           +----Js---+
    +----CW----+      +-Ss-+--TOf-+-Ift-+--PPf-+--------Ost-------+--------R-------+     +----E---+--Pa--+-MVp+   +Ds**v+
    |          |      |    |      |     |      |                  |                |     |        |      |    |   |     |
LEFT-WALL <CAP-MARK> it seems.v to.r have.v been.v Einstein[!<CAPITALIZED-WORDS>] who first.a came.v-d up.a with the idea.n

[linas]

I like the appearing in the parse tree. That is certainly an interesting idea!

But is it superfluous? In almost all cases, if a word attaches with Wd, then it should have been capitalized...(or already was capitalized).

Linking CAP-MARK to the left-wall seems wrong; it really should be linking to the word that is (should be) capitalized. Or better yet, link to both: to the wall, and also to the word that is being capitalized, with both linkages being mandatory, not optional, thus constraining the parses. That makes it similar to the PH link, which indicates a phonetic constraint.

[linas]

This seems wrong:

<CAP-MARK>:     /^[[:upper:]][^'’]*[^[:punct:]]$/fcl

because it violates your earlier guess about French: the lower-case version of E could have been any one of the letters é è ê ë, and all of them should be tried during dictionary lookup. Thus, perhaps the regexes should be of the form:

<REGNAME>: s/match-expr/rewrite-expr/flags;

which 'c' being a possible flag. The 'f' and the 'l' flags would not be needed. If the rewrite-expr has whitespace, it should be split on that whitespace ...

[linas]

Allowing two links between words -- I'm pretty sure this would massively break the existing dicts.

The issue 1,2 for null words - I'm not sure that needs to be solved. If ends up being a null word, that just means that someone capitalized a word that they should not have capitalized.

[ampli]

But is it superfluous? In almost all cases, if a word attaches with Wd, then it should have been capitalized...(or already was capitalized).

I tried to always use the same rule: Provide a CW+ connector on a token after which there is a capitalizable position. But my current implementation is indeed problematic.

Or better yet, link to both: to the wall, and also to the word that is being capitalized, with both linkages being mandatory, not optional, thus constraining the parses.

Of course you are right. I just had a problem how to do it.

One of the ways to start to solve that is having something like:
<CAP-MARK>: (CW- & LLD+) or XXXGIVEN+;
when a downcased word (only) will have LLD-.
This means that the code will need to add LLD- to a downcased word.
I think such a mechanism is generally fine and required when inserting tokens into the sentence, and may have uses in other cases. But I have no idea yet how to define a simple rule for that.

Why I think such disjunct rewrites are not contradicting the current LG theory:
This is as if we had two version for each word in the dict, one is the regular lowercase letter, and one is the uppercased one that has the exact same sentence usage as the lowercase one but that can come in capitalizable positions, thus having an additional mandatory connector.
Instead of heaving these two versions, I just proposed a higher-level approach to programmatically add the needed connector (of course in a general not hard-coded way). This may be needed in cases of "sentence rewrite" and maybe in other cases too.

[ampli]

The 'f' and the 'l' flags would not be needed.

I don't understand how you can do it without an indication like 'f'.
This flag indicates that the regex label should be physically inserted as an additional sentence token.
Or would the rewrite part will imply that such insertion is intended? But in general there may be such rules for other pseudo-morphology uses that don't involve a replacement.

because it violates your earlier guess about French: the lower-case version of E could have been any one of the letters é è ê ë, and all of them should be tried during dictionary lookup.

I cannot see how. I wrote there:

% The 'l' flag signifies that the first letter should be converted to lowercase
% using the TOLOWER affix class, with a tolower() fallback.

So the algorithm is to use a TOLOWER definition if exists, such as
E é è ê ë: TOLOWER+;
and generate all the alternatives (4 in that case), and also make tolower() and generate an alternative.
(in the quoted comment I used the word fallback which is misleading since it is always needed).

<REGNAME>: s/match-expr/rewrite-expr/flags;

I am not sure I understand what would be this rewrite-expr.
Do you mean to have a few tens of such regexes, in the form like:

...
<REGNAME>: s/B(word)/b\1/c
<REGNAME>: s/E(word)/e\1 é\1 è\1 ê\1 ë\1/c
...

(And then there is no tolower() in the code, and an alternative would be generated on white space or end of replacement part.)

[ampli]

Allowing two links between words -- I'm pretty sure this would massively break the existing dicts.

It will be interesting to see where and how...

I always thought multiple links should be allowed.
Else, what can be generally done (in the current scheme) for a token that sometimes has a real meaning of two words, and it cannot even get morphology-split?
(Private cases of such tokens can be solved with pseudo-morphology, like I try to implement now. I encountered this problem in Hebrew with a few one-letter tokens.)

[linas]

I'm trying to look at this issue as a special case or demo for issue #700 Thus,

<REGNAME>: s/match-expr/rewrite-expr/flags;

becomes viable for all sorts of tokenization problems, and not just for capitalization. So, yes, there would be a few dozen rules of the form

...
<DOWNCASE>: s/B(word)/b\1/c
<DOWNCASE>: s/E(word)/e\1 é\1 è\1 ê\1 ë\1/c
...

and no tolower() in the code. The problem with E é è ê ë: TOLOWER+; is that its only good for capitalization, but is not sufficiently generic to solve other tokenization problems (such as splitting units away from numeric expressions.) And so, if you consider splitting away units from numbers, this suggests

<UNITS>: s/([0-9]+)inches/\1 inches/c

which does nothing except insert whitespace between the number and the run-on unit. Then with the power to insert spaces (or insert other text), the need for the f flag is no longer needed.

[ampli]

For splitting units and any other continuous morphology I proposed a better way - defining (in the dict) token boundaries (which side of a token must have whitespace) either by marks on the tokens (similar to the "infixmarks" today) or by link types. It is not hard to to write a tokenizer that will use only this info.

[linas]

programmatically add the needed connector

Yes. I'm not sure how. Ideally, one could say something like "if regex <BLAH> ran, then insert A+ on the first token and insert B- on the second token". Since connectors are order-dependent, one should insert the A+ to the left of, or maybe to the right of all the other + connectors.

In essence, I'm thinking that there is some better way of doing what is done by INFIXMARK+,SANEMORPHISM+ and STEMSUBSCR+.

This is a non-trivial problem: the current Russian dictionaries do seem to be fairly direct and well-structured; I don't want to loose the current simplicity, there. What we currently do is not bad. Yet somehow, the current tokenization mechanism seems inadequate for more complex morphology. In the back of my mind, I'm even thinking of speech-to-text and other discernment problems.

There are other possibilities: for example, instead of a regex with programmatically-added connectors, imagine a neural net component, which took in sentences and spat out tokens and connectors. Viz, did exactly the same thing as a regex+prgram-connectors component did, except that maybe it assigns likelihood values to each possible tokenization. The only reason I'm thinking "neural net" is that it would be trainable -- I do NOT want to hand-write big, complex regex+prog-connector dictionaries. Or perhaps there is a way to auto-convert the NN to a regex+connector dictionary....