Vord paradigm tool

[obelus]

Like any raw data, they need analysis and interpretation.  Examples of the kind of "rule" we might infer from them could look something like this (ignoring spaces):

• In Currier B, a glyph preceded by e is more likely than usual to be followed by y.
  
• In Currier B, d is more likely to be followed by a than by y if it is preceded by y, r, p, n, or l.
  

Quite right, it was cranky of me to fault a descriptive data structure for not delivering concise heuristics.  Patterns emerge as soon as we visualize the raw data in light of a good question, such as the relationships between a glyph's nearest neighbors quoted above.  The pair correlations for a generic glyph's preceding and following neighbors can be calculated from the third-order matrix by aggregating over the second coordinate (summing entries for the middle glyph).  The resulting probabilities for Currier B paragraph text (with spaces) add up to unity, as they must:

   

Rows are labelled by preceding character, columns by following character, in order of decreasing frequency from the upper left.  Pfeaster's e?y rule is a prominently dark, high probability element, and we can read off others relating to word boundaries:

• A character preceded by d is likely to be followed by a space (true enough)
  
• A character preceded by a space is likely to be followed by o or h (makes sense)
  

The general clustering of probability toward the upper left is due in part to the greater density of high-frequency characters per length of text, which increases their chance associations.  This bias is normally removed by dividing each element by its probability in the absence of correlations.  The values then represent the factor by which each pair probability deviates from that of a scrambled text:

   

The e?y rule remains, occurring 4.8 times more often than expected by chance.  It and other deviants (q?k, i?n, y?q) are mostly driven by the highest-frequency trigrams
     dy_     _ qo     edy     in_     che     _ ch     y_q     ey_     qok     aii     iin
Impressive values in the lower right-hand corner are statistical flukes due to low character frequency:  m?s is represented by exactly two matches in the text sample, odam s and pcheam sokedy.

Since the baseline frequency factor factor (unity) appears deep in the blue end, only Russian speakers (or a logarithmic scale) will be able to pick out pairs that avoid the leapfrog relationship.  It looks as if high-frequency glyph i drives away most over-next-door neighbors.

So... once we have framed a rule scheme, we can certainly search for instances in the data, and unexpected ones may turn up.  But can we glean new schemes from the data by some "hypothesis free" method other than trial and error?

(P.S. to a You are not allowed to view links. Register or Login to view.:  Checking a 100x larger sample of fourth-order text, the fraction of 'correct' word tokens converges to 86%.  tolor and qoto only appear in Currier A, right?  Thus they are among the uncounted.)