06-11-2026, 11:08 PM
(06-11-2026, 09:09 PM)quimqu wrote: You are not allowed to view links. Register or Login to view.I put the prohibited pairs in a table, and I marked the ones that [need repair]
Good idea!
I would add more generally
- any token with two or more gallows must be split somewhere between them.
- q<any> where <any> is not o should be repaired to qo, by inserting or replacing the <any>
- ir should be repaired to iin
- m should be expanded to iin
- Ih should be repaired to Ch
- Rh should be repaired to Sh.
- ITh, IKh, IPh, IFh should be repaired to CTh, CKh.CPh, CHh
- any word with n not at the end should be split after the n.
- hh should be repaired to he
- any token that ends in n, in, iin, or iiin and not preceded by a or o should be repaired by inserting a before it.
- that done, any i that is not [preceded by i, a or o and followed by i or n] should be changed to e if the preceding glyph is h, k, t, p, f, or e (repeat this fix as needed).
More rules can be devised. Like in any spelling corrector, there will be a tension between how many errors are fixed and how many correct tokens are broken.
Here is a more general method that could give better results and requires fewer explicit choices. Choose a word structure model that will define what is a 'valid' word. (I propose mine, stay tuned.) Get the vocabulary V (set of all word types that occur) and compute the frequency F(v) of each.
Then take each lexeme (word type) v and enumerate the set X(v) all the strings x that are a few edit steps away from v. For each of these, let P(x|v) be a probability that v has mutated to x because of error at some step. This is (say) 80% if x = v, then decreases according to the number and type of the edit steps needed to turn v into x. The probabilities P(x|v) summed over X(v) must be 1.
Then, for each u,v in V, use Bayes's formula with probabilities F(v) and P(u|v) to estimate the probability P(v|u) that u was actually some other word v that mutated to u.
Hopefully for most words that occur at least a few times we will find that P(v|u) is definitely maximum when v=u; that is, the word u should not be corrected. If the maximum is some v different from u, then we should consider correcting u to v.
Makes sense?
All the best, --stolfi

