[Blog Post] The 490, and other starting character patterns

[Jorge_Stolfi]

Also my statistics only include the 18 line-initial characters, not any amount of random letters. Of the 18 characters, should I compute based on frequency?

If you assume that those 18 characters are equally likely then P(q) = P(o) = P(y) = 1/18 = ~0.056.  Then, if each line-initial letter is just chosen at random, independently, in 1000 lines you would expect 998*(1/18)^3 = 0.18 occurrences of qoy; that is none, or maybe one or two.

But suppose that (say) P(q) = P(o) = P(y) = 0.30, with all the other 15 letters occurring only on 10% of the lines.  Then in 1000 lines, with each initial being chosen at random and independently, you should expect 998*(0.30)^3 = ~27 occurrences of qoy.  

Thus, in order to tell whether the number of qoy occurences is anomalous, you must consider the actual frequencies of the letters in line-initial positions.

And, again, note that some three-glyph sequences will be more common that what the formula says.  For instance, in the second example above, the string qoy may occur only  25 times, but qyo may occur 31 times, oqy 26 times, oyq 33 times...  If you pick the the most common three-letter pattern, that pattern will be more common than expected.

All the best, --jorge