Identifying function words

[MarcoP]

I have computed Zipf's law graphs for the individual sections of the VMS (ZL transcription, ignoring uncertain spaces, only considering Paragraph text). As always, keep in mind that I could have made errors somewhere in the process.

According to Zipf's law, word frequencies should follow the distribution:

P( r )=C/r^alpha

Where:

• P( r ) is the frequency of the word with rank 'r';
  
• C is a constant close to P(1), the frequency of the most frequent word; in English this is typically about 10% (the frequency of 'the');
  
• in the case of English, the exponent alpha is close to 1; so the second word is about half as frequent as the first word; the third word 1/3 as frequent as the first word etc.
  

I have computed the best-fitting values for C and alpha by means of the scipy.optimize.curve_fit python function. As a quality measure, I use Normalized Root Mean Square Deviation (NRMSD), where I normalize by dividing RMSD by the frequency range, nearly identical to P(1).

For comparison, I include a Latin example (the beginning of Virgil's Aeneid) and an English one (the beginning of the Genesis from King James Bible). AstCZ includes the Astrological, Cosmological and Zodiac sections: these have very little paragraph text, so the measures could be particularly unreliable. Sections are ordered as described in the "Rearranging the pages" paragraph of You are not allowed to view links. Register or Login to view..


Graphs:

You are not allowed to view links. Register or Login to view.

   

These are the results I get:
virgil_ NRMSD:0.008  C:0.024  alpha:0.696
genesis NRMSD:0.017  C:0.117  alpha:0.921
1_HerbA NRMSD:0.003  C:0.048  alpha:0.773
2_Pharm NRMSD:0.011  C:0.039  alpha:0.667
3_AstCZ NRMSD:0.027  C:0.026  alpha:0.571
4_HerbB NRMSD:0.022  C:0.030  alpha:0.617
5_Stars NRMSD:0.018  C:0.026  alpha:0.630
6_Bio__ NRMSD:0.020  C:0.046  alpha:0.686

Voynichese has low values for C and alpha (like Latin). The Latin text has a small deviation from the model. King James Genesis has a poor fit (we know from Koen's TTR research that this text is anomalously repetitive).
An interesting (though probably not surprising) thing is that Voynichese results vary a lot, both with respect to the values for C and alpha and the quality of the fit. HerbalA is very close to the model, with a deviation that is only half the already small value of the Latin text. The other Currier-A-section (Pharma) is also very good.
All the Currier-B-sections have high deviations.

The frequencies of the most frequent words in each sample help understanding what is happening (this can also be observed in the graphs):

virgil_  et:3.21% in:1.04% atque:0.55% per:0.55% aut:0.53%
genesis  and:10.71% the:9.71% of:4.56% god:1.69% earth:1.40%
1_HerbA  daiin:4.91% chol:2.56% chor:1.84% s:1.36% shol:1.25%
2_Pharm  daiin:4.06% chol:1.75% cheol:1.52% okeol:1.43% ol:1.38%
3_AstCZ  aiin:1.82% or:1.76% ar:1.76% daiin:1.65% ol:1.20%
4_HerbB  daiin:2.17% chedy:1.74% or:1.71% chdy:1.50% aiin:1.50%
5_Stars  chedy:1.77% qokeey:1.55% qokeedy:1.32% aiin:1.23% qokaiin:1.18%
6_Bio__  shedy:3.31% chedy:2.82% qokedy:2.54% qokain:2.54% ol:2.50%

The Bio section is particularly clear, showing that the frequencies of the five most common words are almost "flat": "shedy" is comparable with the frequency of "et" in the Latin text, but the fifth word "ol" is about five times as frequent as the Latin fifth word "aut". This non-Zipf-like flatness of the left side of the curve can also be appreciated visually in the bottom four graphs.

Even more puzzling, the five most frequent words are completely different between A and B. In principle, one would expect all of the five most frequent words to be function words, but in Voynichese this does not seem to be the case (this problem has often been mentioned by Torsten).

If Voynichese is meaningful and its words correspond to words in a natural language, I can only think of these possibilities:

• the underlying language has no function words (I am not sure this is really possible);
  
• the same function words in the "source" language are represented by different words in the different  sections.