[Blog Post] The 490, and other starting character patterns

[Jorge_Stolfi]

if the line-initial letters had been chosen randomly and independently with those frequencies

Two random variables X and Y are independent, by definition, if Prob(X=x and Y=y) = Prob(X=x)Prob(Y=y) for any possible values x,y

If each letter S[k] in the sequence is independent from all the others, in particular it must be independent from the previous one S[k-1].  You can test this condition by computing the frequency P(xy) of S[k-1]S[k] = xy and comparing it with the product of frequencies P(x)P(y).  For example, since P(q) = 0.1286 and P(o) = 0.1317, the frequency P(qo) of qo in consecutive positions should be close to 0.1286*0.1317 = 0.0169.  That is, P(qo)/(P(q)P(o)) should be close to 1.

However the frequencies are only an approximation to the probabilities, and the error is large when the letters and letter pairs occur only a few times in the sample.  So this test should be applied only to the most common letters, like the top five in your list.

Also, this test only checks whether each letter is independent from the previous  one.  If the line-initial letters pass this test, there may still be more complicated dependencies.  Like "S[k] is equal to S[k-3] 70% of the time, unless S[k-7] is q, in which case ..."

Also, if one computes a large number of statistics about something, some of the statistics will be "anomalous" just by chance.   Like, if you ask 1024 financial gurus in the morning whether a certain stock will go up or down during the day, for 10 consecutive days, there is a good chance that one of them will be correct all 10 times.  It does not follow that her predictions are better than the others.  Or better than flipping a coin...

All the best, --jorge