12-05-2026, 08:02 AM
At the cost of sounding pretentious, please let me give some advice to all those who are studying the text.
When I was a PhD student at Stanford, I met a fellow Computational Geometer from Austria, who was absolutely shocked when he learned that, in the almost 10 years I had been there, I had never been to the famous Stanford Football Stadium to watch a game by the famous Stanford Football Team. That, in fact, I had never watched a game of American football, not even on TV. So he demanded that we go watch the next game there.
I agreed to buy the tickets; but, since I had never done that, I choose what I thought were good seats -- right next to the field.
So we watched a full game of One-Dimensional Football -- where the field was just a green horizontal line, with white blobs running back and forth along it for some inscrutable reason.
So here is the advice: don't waste your time computing character-based statistics. Such as character and n-graph frequencies, character and n-graph entropy, vertical or horizontal character correlations, etc.
The reason is the same reason for which you should not buy front-row tickets to a football or soccer game. Namely, what you will get is a projection of the full text statistics onto a very small numeric space. This projection will almost certainly mix up many distinct meaningful features of the text; and you will never be able to untangle that mess by staring at all the tables and plots.
Like, the frequency of "h" in English is the sum of the frequency of the "h" sound and the frequencies of digraphs "ch", "sh", "th", "ph", "rh", etc; and the frequencies of these sounds and digraphs were defined by multiple distinct historical and linguistic factors. You will never be able to separate and understand these factors by studying letter and n-gram statistics.
For one thing, all character-based statistics are highly dependent on the spelling system and transcription alphabet. For instance, one could easily double or halve the entropy per character of a text by a trivial change of spelling (not even encryption), such as using digraphs instead of single letters for common sounds, or encoding the same sound in several different ways, randomly chosen.
Apart from that, character-based statistics are highly dependent on the nature and topic of the text. There is no such thing as "the frequency of 'e' in English" or "the most common digraph in Latin". The statistics of a character or n-gram are determined by the statistics of the most common words that contain it; and word statistics -- even of "function" words like "the", "of", "and", "is" -- are strongly dependent on topic.
After all, when one is writing, one is constantly choosing words -- not characters.
In a booklet that explains how to make sixty six luxury boxes for mixed toxic wax halluxes with an ax, label them with Roman numerals, and then relax, character-based statistics will reveal an abnormally high frequency of the letter "x". Further investigation may reveal a puzzling absence of the digraph "ex" in spite of "e" and "x" being very common. But what conclusions could one draw from such data?
One may easily draw wrong conclusions, like that the pamphlet is in an original "language A" (dialect, encoding, etc), clearly distinct from the "language B" of that other document that advises against having sex in your vexillologist ex's triplex in Texas while texting an explanation of how to extract the index of a vertex in a convex cell complex.
A concrete example of misdirection due to character statistics is the observation that, on the VMS, words that begin with qo are more common at line-start than elsewhere. Now we know that one cause of that anomaly has nothing to do with those characters: it is just the bias towards longer words in that position that is created by the process of breaking text into right-justified lines -- and it so happens that qo-words are longer than average. (This side effect of line breaking may not be enough to explain all the anomalies of qo-words; but it is still not known where there is anything beyond it.)
In conclusion: if you must do statistical analysis of the text, don't focus on characters, focus on words. And if you need to merge words into word classes (say, to reduce the volume of the results or the sampling noise), try to use classes that are defined by some semantic criterion (like co-occurrence), not by the characters that occur in them (like "qo-words" or "words with gallows").
All the best, --stolfi
PS. My Austrian friend was of course furious about my seat choice, and demanded a replay, with him buying the tickets that time.
So we watched the next football game at the Stanford Stadium from a place with superb view of the field.
At night.
A cold night.
A cold and rainy night.
But I don't know what VMS-relevant advice I could draw from this part of the story. "Don't let an Austrian friend decide which statistics you should compute?" Duhh ...
When I was a PhD student at Stanford, I met a fellow Computational Geometer from Austria, who was absolutely shocked when he learned that, in the almost 10 years I had been there, I had never been to the famous Stanford Football Stadium to watch a game by the famous Stanford Football Team. That, in fact, I had never watched a game of American football, not even on TV. So he demanded that we go watch the next game there.
I agreed to buy the tickets; but, since I had never done that, I choose what I thought were good seats -- right next to the field.
So we watched a full game of One-Dimensional Football -- where the field was just a green horizontal line, with white blobs running back and forth along it for some inscrutable reason.
So here is the advice: don't waste your time computing character-based statistics. Such as character and n-graph frequencies, character and n-graph entropy, vertical or horizontal character correlations, etc.
The reason is the same reason for which you should not buy front-row tickets to a football or soccer game. Namely, what you will get is a projection of the full text statistics onto a very small numeric space. This projection will almost certainly mix up many distinct meaningful features of the text; and you will never be able to untangle that mess by staring at all the tables and plots.
Like, the frequency of "h" in English is the sum of the frequency of the "h" sound and the frequencies of digraphs "ch", "sh", "th", "ph", "rh", etc; and the frequencies of these sounds and digraphs were defined by multiple distinct historical and linguistic factors. You will never be able to separate and understand these factors by studying letter and n-gram statistics.
For one thing, all character-based statistics are highly dependent on the spelling system and transcription alphabet. For instance, one could easily double or halve the entropy per character of a text by a trivial change of spelling (not even encryption), such as using digraphs instead of single letters for common sounds, or encoding the same sound in several different ways, randomly chosen.
Apart from that, character-based statistics are highly dependent on the nature and topic of the text. There is no such thing as "the frequency of 'e' in English" or "the most common digraph in Latin". The statistics of a character or n-gram are determined by the statistics of the most common words that contain it; and word statistics -- even of "function" words like "the", "of", "and", "is" -- are strongly dependent on topic.
After all, when one is writing, one is constantly choosing words -- not characters.
In a booklet that explains how to make sixty six luxury boxes for mixed toxic wax halluxes with an ax, label them with Roman numerals, and then relax, character-based statistics will reveal an abnormally high frequency of the letter "x". Further investigation may reveal a puzzling absence of the digraph "ex" in spite of "e" and "x" being very common. But what conclusions could one draw from such data?
One may easily draw wrong conclusions, like that the pamphlet is in an original "language A" (dialect, encoding, etc), clearly distinct from the "language B" of that other document that advises against having sex in your vexillologist ex's triplex in Texas while texting an explanation of how to extract the index of a vertex in a convex cell complex.
A concrete example of misdirection due to character statistics is the observation that, on the VMS, words that begin with qo are more common at line-start than elsewhere. Now we know that one cause of that anomaly has nothing to do with those characters: it is just the bias towards longer words in that position that is created by the process of breaking text into right-justified lines -- and it so happens that qo-words are longer than average. (This side effect of line breaking may not be enough to explain all the anomalies of qo-words; but it is still not known where there is anything beyond it.)
In conclusion: if you must do statistical analysis of the text, don't focus on characters, focus on words. And if you need to merge words into word classes (say, to reduce the volume of the results or the sampling noise), try to use classes that are defined by some semantic criterion (like co-occurrence), not by the characters that occur in them (like "qo-words" or "words with gallows").
All the best, --stolfi
PS. My Austrian friend was of course furious about my seat choice, and demanded a replay, with him buying the tickets that time.
So we watched the next football game at the Stanford Stadium from a place with superb view of the field.
At night.
A cold night.
A cold and rainy night.
But I don't know what VMS-relevant advice I could draw from this part of the story. "Don't let an Austrian friend decide which statistics you should compute?" Duhh ...
![[+]](https://www.voynich.ninja/images/collapse_collapsed.png)