New Voynich text analysis/visualisation tool

[Jorge_Stolfi]

character and n-gram statistics are more confusing than illuminating

Can you point to a scenario in which such statistics would inherently yield a misleading result, as opposed to one that's just challenging to explain, or that runs contrary to expectations, or that isn't a good fit for a particular hypothesis?

Suppose you have a herbal written all in English by the same person, but it is divided in two sections, one covering grasses and one covering bushes. Digraph statistics may detect that the 'rb' digram is rather common in the first half but nearly absent in the second half, while the opposite is true for the digrams 'bu', 'ba', and 'rk'. This may lead Martian cryptologists (who can't read English) to conclude that the two sections are in different languages or are written by two different people.  But a word analysis would show that those differences are due entirely to the word 'herb' being heavily used but only in the first section, while the words 'bush' and 'bark' are heavily used but only in the second section.

Or suppose that the Martians analyze digraph statistics of an "encrypted" book as a function of position line, and notice that the statistics of 'th' are lower than average within the first 10 letters and higher than average in the last 10 letters of each line.  They may conclude that the "encryption algorithm" is restarted at the beginning of each line, and that the end of each line is padded with random "filler" words.  But in fact the anomaly is due to the fact that the trivial line breaking algorithm causes the first word of each line to be longer than average, and the last 2-3 words to be shorter -- and the frequency of 'th' in English texts is dominated by its occurrence in many common short words.

Trying to understand an unreadable text by looking at character and digraph statistics is like trying to understand the ecology of different regions by counting animals with tails and without tails.  The counts will lump aardvarks with alligators, bears with beetles, caterpillars with chimpanzees, ... The counts will probably show that the Sahara is somewhat different than the Amazon --- but not much beyond that.  One may stare at those numbers for a whole life and never understand why they are different.

Sure, character and digraph statistics are the first thing one should do when presented with an "encrypted" text.  They can give  useful clues, e. g. whether it is a simple substitution cipher or something more complicated, like a Vigenère or codebook cipher; and, in the first case, they will suggest some letter assignments, for each possible language.  But they cannot help much if the text turns out to be in an unknown language like Etruscan or proto-Elamite or the Rohonc script.  To make progress in those cases one must look at words, not just characters...

All the best, --stolfi