"The Currier languages revisited" revisited

[Jorge_Stolfi]

Re the nature of the difference between Languages A and B. I suspect that may people have made the following observation already, but here it is again anyway. 

This is the plot of the relative frequencies frA(w), frB(w) of the most common words in Herbal-A and Herbal-B.  (The plot only shows words that have frequency >= 0.5% in both sets):
   
And here is the same plot when each word in either file is mapped to its "root", as defined below:
   

The mapping from word to "root" entails the following substitutions:

• Replace all weirdos by '?'
  
• Delete all instances of q o a y.
  
• Map all instances of k t p f K T P F to f. (Why not t? Because You are not allowed to view links. Register or Login to view. would become t17v... )
  
• Map all instances of d l r s to r.
  
• Replace Ih by Ch and Ifh by Cfh
  
• But then replace all instances of C or h by e
  
• And then delete all instances of e i n m g 
  

Words that become empty are replaced by a word with a single '?'.  

These rules were motivated partly by guesses at what are the "important" letters, and partly by observation that some pairs (like o and a, r and s, Ch and ee) are often swapped by the Scribe, eventual Retracers, and the Transcribers.

The frequencies here are computed with the algorithm I described earlier in some other thread, where uncertain spaces ',' are treated as '.' or no-space with 50% probability, independently.  So that in ky.Cho,d,Shy.ol the words ky and ol count as 1, Cho and Shy count as 0.5 each, and Chod d dShy ChodShy count as 0.25 each.

The transcription is a merge of my own new partial transcription from the BL 2014 images with Rene's IVTFF transcription.  The processing above considers only paragraph text (excluding "labels" and "titles") of pages that are definitely "herbal" (thus excluding f1r).

Discussion: I interpret this result as evidence that Herbal-A and Herbal-B are not really different languages, but the same language spelled or encoded in fairly different ways.  The differences would involve different usage of the letters that are deleted by the rules above, and possibly different choices between alternatives that are mapped to the same letter.

All the best, --stolfi

PS: the "fr«" in the second plot is a bug in my script. Please disregard.  It should be counted as just "fr"