The 'Chinese' Theory: For and Against

[Jorge_Stolfi]

JoJo, forgive me for repeating the advice I gave on another thread:  character-level statistical analyses (character and digraph frequencies, entropies, correlations, etc.) are very likely to be a waste of time.  

One of my character-level statistical analyses led you to remove a wrong criterion for identifying paragraph breaks in the Stars section.  Not a total waste of time, surely?  And when you draw conclusions about ... p is k on the top row, is that really based on a word-level analysis and not a character-level one?  It's when one goes beyond the individual glyph and look at clusters, if not full word types, that problems with glyph mapping tend to emerge.  And I say that as someone who has feverishly mapped virtually every glyph to another at some point.

Apologies for coming off as too negative.  (But please note that I said "very likely",  not "guaranteed".)  

Character-level statistics do indeed have utility in the right contexts.  I have used them myself when developing my word structure model, which includes the proposal that p/f are "capital" forms of ke/te (or possibly more; and still not sure which is which).  And, of course, when they do yield useful insights, it would be silly to dismiss these just because they are "character-based".

But I still stand by my advice.  It is not a light comment: it comes from 30 years of watching people spend inordinate amounts of time doing character statistics that did not yield any useful conclusions.  

Or, worse, that led to wrong conclusions.  Like the attempts to identify the language (or prove that it is not a natural language) by comparing character entropies of Voynichese to those of sample texts.

Many people seem unaware that character statistics are not diagnostic properties of language, topic, or author.  They do depend on those attributes, in the sense that they will usually give different values on plaintexts that differ in those attributes.  However, the dependency is quite complicated and indirect, and character statistics are obliterated by even small changes in the spelling or encryption. 

Character statistics can lead to cracking of a simple substitution cipher applied to a plaintext with mostly "official" orthography.  But even slight complexifications -- like context- of position-dependent nulls and one-to-many letter encodings -- will render that tool useless.  (IIUC, this was the case of the Copiale cipher.)  

As I noted before, if the words of a text have very unequal frequencies (as implied by Zipf's law), the frequency of a character or digraph in a text is largely determined by its occurrence or absence in the most common words.  The assertion that "th" is more common at the start of an English word than expected by chance may be true, but it is not very informative.  It is even somewhat misleading, because it seems to suggest that "th" is sort of a fungus that grows indiscriminately on the edges of words on islands with rainy climate and warm beer.  The statement some of the most common English words begin with "th", while technically weaker, should give a better idea of what is going on.

Many others have noted that character and digraph frequencies around line breaks are different from those in the middle parts of lines. @JoJo recently pointed out anomalous statistics about initial Sh, Ch, and q. @DG97EEB posted on patterns of the initial letters of successive lines. These anomalies are real, but they are not  properties of those characters.  They are properties of certain words and sets of words, which mostly happen to use those characters.

So, if you happen to notice anomalies in character or n-gram statistics, you should not stop there, or try to deduce character-based "rules" from that data.  You should check which words are responsible for the anomaly, and try to reformulate the observation in terms of words.  

It is likely that you will find that (say) it is not that q is more common than expected at line start, but rather that the words qokeedy and okeedy and keedy are more enhanced there.  Or that words with the qo prefix tend to be longer than those without it -- and line breaking naturally occurs more often before longer words.

about final m replacing final n

Character statistics indeed suggested that m could be an abbreviation.  But the theory that it is an abbreviation for iin (not n) came mainly from observing that the word dam, specifically, was very common at the end of lines -- suggesting that it was an abbreviation for the word daiin, which is common outside of that context.

Same  for my current belief that final ir is mostly  a scribal error for iin.  I first suspected it from the frequencies of the  final "elements" iin, iiin, ir, and iir in my word structure model.  But it became more certain when I saw the words dair and kaiin (but not taiin!) show up often in the SBJ-SPS matchings where daiin was expected.  Then I realized that, in moderately sloppy handwriting, the pair in could be easily be confused with r, and d could be confused with k (but not with t!) 

All the best, --stolfi