Heap's Law

[davidjackson]

ReneZ: To characterise the text in this manner, what one could do is take a group of words (say 1000) anywhere in the MS, and then inspect the following words (a smaller group, e.g. 100) to see how many of them occur in the previous 1000 and how many are new. One can do this in two different ways, either counting all words of the 100, or only the unique words of the 100.

Or, possibly, by comparing blocks of text to one another - either page by page, or even paragraph by paragraph. This could be a way to identify "subjects" within the corpus. Sudden bursts of new words would suggest that a topic change has been introduced - something the larger scale graphs that I have posted above seem to show.

Another use could be to find blocks of "null filler" text, which some people have suggested exists in the corpus. If we identify lines of text that appear to be filled with new unique tokens, then this could be null cover text designed to bulk out the real content.

MarcoP - While copying already generated words is an excellent way to approximate Heap's law, one must also find a way to appropriately constrain the generation of new words.

Indeed. One must of course assume that power laws were unknown to the scribes, and so such a constraint must make sense for the imagined mindset of the scribe. More to the point - why constrain the text in this way? It makes no sense within our imagined knowledge of the time when the text was written.

I haven't had time to examine Fisk's work yet, I will try to later this week.

The generated text only contains words copied from each other.

Torsten - this is certainly apparently true. However, I feel that your theory, which otherwise appears to account for much of the way the text was written, fails to explain the grammatical structure which appears to constrain Voynichese (see Stolfi's You are not allowed to view links. Register or Login to view.paradigm). What exactly are these structures, and why do they apply throughout the manuscript? Because if you could link those structures into your generation algorithm, we would be some way closed towards proving your theory. And why was the text created in such a way that unique words appear to be introduced in a topic like way? (OK, I don't expect you to prove that second question!   )