Thank you Bernd and Marco for keeping an eye on this. Everything I know about statistics I specifically learned for this research, so all of this is very experimental for me, and likely to be suboptimal.
Anyway, I don't have much time today, but I will simply share my data: You are not allowed to view links. Register or Login to view.
You will find three sheets. The first one is simply raw entropy data. This is the cleaned, updated corpus, so it is to be preferred over the one I shared years ago.
The second sheet is the data, normalized to the corpus.
The third sheet is something I was working on for fun. It is the data normalized to shuffled texts.
For this third tab, what I thought was the following: imagine a scenario where we put a regular text into a bag and shake it up, then pull out one word at a time and form a text like that. What would happen if call this scenario "normal", and see how actual meaningful texts compare. What is the impact of the rules of natural language and text composition on TTR?
To test this I took a number of texts and shuffled the words. I took mostly texts with an average overall TTR to aim for a situation as average as possible. I then normalized to these shuffled texts. Now, with what Marco said about shuffling actually lowering small-window TTR (the the, this this....), one hypothesis might be that normalizing to shuffled texts will bring Voynichese more in line with the norm. In this case, we can expect Voynichese to form a line parallel with 0. This is a quick graph I made with Voynich samples and some random other texts:
Focus first on the blue and teal lines, Herbal B and Q20. I would have expected them to behave more like the norm, but not this much!
The yellow line, Herbal A, is still relatively parallel but with lower overall TTR. Q13 is lower still.
Now what I did here is weird, using the average and stdev of one set to plot another. Especially stdev will have an impact on the size of the undulations in the lines. So this could certainly be optimized. However, apart from that, I do think this is a meaningful result: Voynichese distributes its vocabulary as if it were drawn from a bag, at least it does so more than regular texts. In the current experiment, this is especially prominent for the B-language. A-language samples are less clear due to their inherently lower TTR.
Anyway, I don't have much time today, but I will simply share my data: You are not allowed to view links. Register or Login to view.
You will find three sheets. The first one is simply raw entropy data. This is the cleaned, updated corpus, so it is to be preferred over the one I shared years ago.
The second sheet is the data, normalized to the corpus.
The third sheet is something I was working on for fun. It is the data normalized to shuffled texts.
For this third tab, what I thought was the following: imagine a scenario where we put a regular text into a bag and shake it up, then pull out one word at a time and form a text like that. What would happen if call this scenario "normal", and see how actual meaningful texts compare. What is the impact of the rules of natural language and text composition on TTR?
To test this I took a number of texts and shuffled the words. I took mostly texts with an average overall TTR to aim for a situation as average as possible. I then normalized to these shuffled texts. Now, with what Marco said about shuffling actually lowering small-window TTR (the the, this this....), one hypothesis might be that normalizing to shuffled texts will bring Voynichese more in line with the norm. In this case, we can expect Voynichese to form a line parallel with 0. This is a quick graph I made with Voynich samples and some random other texts:
Focus first on the blue and teal lines, Herbal B and Q20. I would have expected them to behave more like the norm, but not this much!
The yellow line, Herbal A, is still relatively parallel but with lower overall TTR. Q13 is lower still.
Now what I did here is weird, using the average and stdev of one set to plot another. Especially stdev will have an impact on the size of the undulations in the lines. So this could certainly be optimized. However, apart from that, I do think this is a meaningful result: Voynichese distributes its vocabulary as if it were drawn from a bag, at least it does so more than regular texts. In the current experiment, this is especially prominent for the B-language. A-language samples are less clear due to their inherently lower TTR.
![[+]](https://www.voynich.ninja/images/collapse_collapsed.png)