The Voynich Ninja

(01-07-2026, 10:29 AM)nablator Wrote: You are not allowed to view links. Register or Login to view.metric                      REAL        GENERATED
----------------------------------------------------
char H2 (bits)              2.02        2.02
char H3 (bits)              1.75        1.78

These two coincidences are guaranteed for a Markov of order 3.  In fact the output should reproduce the k-gram frequencies of the training text for k up to 4; and the entropies H0 to H3 are determined by those frequencies.

Quote:word len mean               5.19        5.20

This coincidence too is expected because the frequency of [SPACE] is guaranteed to be the same, and the average token length is the reciprocal of that.

Quote:type/token ratio            0.281       0.271
hapax % of types           68.8        67.3

If the output text follows Zipf's law, these numbers depend only on the number of tokens in the text.  I vaguely remember claims that the output of Markov models do follow Zipf's law.

Quote:word len var/mean           0.63        0.75

I don't know what is expected for this metric

Quote:self-citation               0.676       0.648

To properly train a Markov of order 3 with an alphabet of size N one needs a training text with several times N^4 letters, because it must include every possible letter that may follow each possible 3-gram with non-negligible probability.  If the training text is too short, the transition table will often show a single choice for the next letter after the current 3-gram.  In that case the model will often repeat long phrases of the training set, much longer than 3-4 letters.  This may be the explanation for these numbers being similar.

Quote:immediate-repeat %          0.72        0.47

On the other hand, an order-3 Markov should often "forget" which word it produced before the current one.  Thus, if the training set had a higher than normal frequency of immediate repeats, the synthetic output will probably fail to reproduce that particular feature.

Quote:suffix(2) profile cosine real↔gen: 1.000  (1.0 = same inventory)
prefix(2) profile cosine real↔gen: 1.000

If the prefix and suffix have up to 4 characters, this metric is merely a consequence of the model reprtoducing the same 4-gram frequencies of the training set.

All the best, --stolfi