[Blog Post] A Universal Template

[obelus]

what is the most likely valid word that is not attested?

One approach to this question could be to synthesize a large text sample according to some language model, then examine the population of unattested words.

We can simulate text statistically by sampling the character correlation matrix.  The largest that I can comfortably handle is fifth order;  that is, the probability of each succeeding character is determined by the preceding 4-mer (for Bennett 1976, a "Fifth-Order Monkey").  Using all paragraph text from Takahashi IT2a-n.txt as input, typical output rambles along like this:

ar sheey qokar qokaiir ychey choltaiir chody shol daiin qokaiin y ycheey cheey qokeeo ypchd ar lkain qopar okaiin okal  okedy qotaly s oteey roiir cthol dol qokeeo chockhey qokalchy olpchedy ysheedy qotchy dor chey qokai qokeeedy qokaiin chckhol daiin yteedy cheor aiin qokal kol cheody chey pshol cheky cthes qopchedy ol sheey teey okeoly olchey  okeeolkaiin qoaiin ychedy pcheey shkeody qokal keeain cheol qotaiin yteey raiiin chedy qokaldy sairy poeeady  pchey cthey dchckheody qotedy cphy oteodaiin...

When such a text of one million EVA characters is parsed into words, 90.% of the tokens are attested in the source.  The exact frequencies change from run to run because the matrix is sampled pseudorandomly.  But some words are common near the top of the most-frequent-but-unattested list:

orchy
shodar
chokchdy
qotcheey
lchedaiin
lchd
lches
ychaiin
chotchor
qoksho

Each of these represents about 0.2% of the unattested tokens generated, or ~0.02% of the synthetic text as a whole.  Thus the statistical model will produce several instances of orchy in a sample of 35 000 words.  The predicted frequency does not trivially account for its absence in the source.

The "model" here is a brute statistical description of the local EVA character sequence, so there is nothing to be learned by peeking under the hood.  Could there still be interesting patterns in the population of possibly "unattested valid" words of this type?