Identifying function words

[MarcoP]

I recently read an interesting paper about function words: You are not allowed to view links. Register or Login to view., by Tony C. Smith & Ian H. Witten, 1993

The largely quantitative approach of that paper made me curious to experiment with function words and the MATTR technique that Koen brought to the attention of the Voynich community.

I considered the first 1000 word tokens from a number of texts and manually partitioned all word types into the two categories of FunctionWords and ContentWords.

These are the part of speech classes I treated as Function Words:

• articles (e.g. the)
  
• prepositions (e.g. in)
  
• pronouns (e.g. you)
  
• possessive determiners (e.g. your)
  
• conjunctions (e.g. and)
  
• auxiliary verbs (e.g.  have)
  
• adverbs of negation (e.g. not)
  

Of course, I tried to be consistent, but it's likely I made errors.

These are the 10 texts I examined (I only provide links for those that are not easy to find):

• en_gen King James Genesis, English, XVII Century
  
• en_herb You are not allowed to view links. Register or Login to view., English, XVI Century
  
• gr_gen Greek Genesis, III Century BCE
  
• oc_thes You are not allowed to view links. Register or Login to view., Occitan, XIV or XV Century?
  
• it_mach Machiavelli Il Principe, Italian, XVI Century
  
• lat_bon_n You are not allowed to view links. Register or Login to view., Latin, XIII Century, normalized transcription
  
• lat_bon_d Bonaventura, Soliloquium, Latin, diplomatic transcription
  
• lat_gen Latin Vulgate Genesis, IV Century
  
• lat_matt You are not allowed to view links. Register or Login to view., Latin, XVI Century
  
• lat_vir Virgil, Aeneid, Latin, I Century
  

This table shows the 20 most frequent word types in each text with my tentative Function(green) / Content(yellow) classification.

   

The interesting result is that there is a significant positive correlation (0.91) between MATTR and the % of content tokens. After I ran these experiments, I found that this percentage is known as Lexical Density.

The correlation is possibly explained by the fact that inflected languages result in higher MATTR (since inflection produces more word types from each stem); these languages use fewer prepositions, since such information is sometimes carried by noun cases. Latin is about as inflected as Greek but additionally it has no articles, producing an even higher Density.


If one assumes that Voynichese words correspond to words in a language that is somehow comparable with the European languages I examined, it is possible to infer a rough estimate for Lexical Density in the VMS. MATTR (I used a 500 words window) has some uncertainty due both to differences between Currier A and B and the presence of uncertain spaces (I used the Zandbergen-Landini transliteration). Some additional uncertainty is due to the fact that the correlation with Density is not perfect, e.g. the Italian and Occitan texts have similar MATTR values and Lexical Density values about 12% apart. Overall, I would say that Density for the VMS could fall into the 55-75% range, i.e. 25-45% of word tokens can be expected to be function words.

Of course, it is possible that Voynichese words do not correspond to words in any language, or that the underlying language is so different from the languages I considered that it does not agree with the correlation discussed here.

NB: VMS dots in the plot show actual MATTR values, while I manually selected Y values to fall near the regression line.