Vord paradigm tool

[pfeaster]

The 'leapfrog' pattern from my previous post is unusual because it can easily be implemented as a functional operator on the big correlation matrix.  Other schemes, of greater linguistic merit, may be roundabout or impossible (for me!) to check in that way.

There are a couple of different ways to analyze "leapfrog" patterns (I like that term!), and I believe they have different implications.  

One is to work out matrices for n-grams beginning with each glyph, such as [o*], [o**], [o***], etc., where each * is a single wildcard glyph: thus, what percentage of the time is [o***] followed by [y], as in [otedy]. 

An alternative is to compare specific values of [o*], [o**], [o***] against the matrices for specific corresponding sequences [*], [**], [***].  So, for example, [oted] makes up some percentage of tokens of [ted].  If the [o] had no statistical impact on the glyph that follows after [oted], we would expect the frequency of [otedy] to be roughly that same percentage multiplied by the frequency of [tedy].  Not exactly, of course; the data is likely to be rather noisy.  But that's how things ought to trend.

The first approach shows how likely one glyph is to appear a certain number of spaces ahead of another glyph, without factoring in the intervening glyphs.  This will tend to expose the same kinds of patterns as a "slot-based" word morphology.  Thus, [o***] might commonly be followed by [y] just because [o] and [y] commonly occupy "slots" that are separated by that many positions and the intervening slots are commonly filled rather than empty.

The second approach will instead show how the presence of the first glyph affects what usually happens after a specific given sequence of intervening glyphs.  This should factor out many of the strictly "slot-based" patterns and let us detect other kinds of correlation over a distance.  Thus: [y] may be common after [ted], but admitting that to be so, is it then more or less common than usual when [o] appears beforehand?

The results of the second approach can also be generalized by aggregating the statistics for all possible values of, say, [o***] against [***], examined case by case, to see if there are any consistent patterns.

The sample "rules" I listed earlier were worked out using the second approach.

So, for example, it's not just that the glyph four, five, or six places ahead of one [t] is more likely than average to be another [t].  That could be explained in terms of average word lengths and a "slot-based" word morphology.  Rather, it's that the glyph four, five, or six places ahead of one [t] is even more likely to be another [t] than it should be based on aggregate matrices for intervening glyph sequences.

Whether this sort of pattern explains or is explained by a tendency of similar words to recur near each other, I don't know!