Automated Topic Analysis of the Voynich Manuscript

[Jorge_Stolfi]

Just to clarify: in your analysis you take the parags (only, excluding labels and titles) of each page, do some cleanup, compute the token frequency distribution in those parags for each page, and then compare the distributions of different pages to identify "topics", by various criteria.  Is that so?

You may be interested in this python code that I wrote to deal with uncertain word spaces.  

• You are not allowed to view links. Register or Login to view.
  
• You are not allowed to view links. Register or Login to view.
  
• You are not allowed to view links. Register or Login to view.
  

Basically the script assumes that each comma is either a word space (like a period) or no space with some probabilities P and 1-P, say 50-50.  Then it enumerates the possibilities and outputs all possible words with a fractional count.  

For instance, in the line Che,y,ky.Cho.dal,dar, it would output the following counts

• 0.50 Che
  
• 0.25 Chey
  
• 0.25 y
  
• 0.25 yky
  
• 0.25 Cheyky
  
• 0.50 ky
  
• 1.00 Cho
  
• 0.50 dal
  
• 0.50 daldar
  
• 0.50 dar
  

Note that the output lines that contain a given input glyph must add to 1.

This is still far from the "correct" freq distr, since it cannot account for uncertain spaces that were omitted in the transcription, or transcribed with ".".  But hopefully it is better that treating all commas as periods (P=1), or deleting all commas (P=0).

Beware that my "IVTFF" files are not yet quite legal IVTFF as defined byt Rene), so you may need to do some adjusting in order to read his files.

The same principle should be used when computing word pair frequencies.
Namely, from that line we should get

• 0.250 Che.y
  
• 0.250 Che.yky
  
• 0.250 Chey.ky
  
• 0.250 Cheyky.Cho
  
• 0.250 y.ky
  
• 0.250 yky.Cho
  
• 0.500 ky.Cho
  
• 0.500 Cho.dal
  
• 0.500 Cho.daldar
  
• 0.500 dal.dar
  

Here the lines that contain each input possible-word must add to 2; except the first and last possible-word,which must add to 1.

But my code does not do this yet.

All the best. --jorge