Word Entropy

[MarcoP]

From Bowern and Lindemann papers, I understand that word stats in the VMS appear to be closer to written languages than character stats. I recently had another look at what Koen did with word entropy.
Using his corpus, I successfully produced a graph similar to You are not allowed to view links. Register or Login to view. (word-entropy vs conditional word entropy). A minor difference from what he did is that I considered the first 10k words for each file (Koen used the first 5k). The green diamonds are Voynich samples.

   

I then observed that plotting word-entropy vs bigram (biword?) entropy isolates the Voynich samples even more. For a given biword-entropy value, Voynich samples have lower word entropy than language files from Koen's corpus. For instance, compare the Takahashi transcription (04.TT) with  the Latin Aelredus file:
04.TT_ivtf_v0a H1: 9.73244 H2: 13.05730
Lat_ Aelredus H1: 10.22625 H2: 13.04832

The values for H2 are nearly identical, but H1 is much lower for the VMS (plot "zoomed" on the relevant interval).

   

How can one transform the Latin file so that biword entropy is basically unaffected while word entropy drops to values closer to the VMS? A possible way to achieve this a many-to-one mapping of some word types, so that the lexicon is reduced (hence word entropy dicreases) but most biwords are still distinct (e.g. because only one of the two words in a biword is typically affected).
I ran several experiments and something that appears to work decently in this case is this regex transformation:

sed -e 's/\([a-z]\)[a-z][a-z][a-z][a-z][a-z][a-z]*\([a-z][a-z]\)/\1q\2/'

The transformation only affects words that are at least 8 characters long. The first character and the last two characters of long words are preserved, while the "core" of the word is replaced by an arbitrary fixed character ('q' in this case).

These are the first 20 words in the original file:
caput primum libri hujus scribendi occasio cum adhuc puer essem in scholis et sociorum meorum me gratia plurimum delectaret et

And the transformed version:
caput primum libri hujus sqdi occasio cum adhuc puer essem in scholis et squm meorum me gratia pqum dqet et

For instance, in the text words like 'delectaret', 'dulcesceret', 'disputaret', 'displicet' are all mapped to 'dqet'

As the plot shows, values for the transformed Aelredus file are quite close to the curve joining the different VMS samples. These are the measures for word-entropy and biword entropy on the transformed file:
04.TT_ivtf_v0a H1: 9.73244 H2: 13.05730
Lat_ Aelredus H1: 10.22625 H2: 13.04832
aelr.q.transf. H1: 9.56466 H2: 13.00754


This is not a good model for Voynichese, since I don't think it can result in a particularly high character entropy. But in my opinion this transformation could support the idea that Voynichese includes 'homophones' (or homographs?) i.e. conceptually distinct words that are written identically.