The structure of the Voynich text and how it may be generated

[Jorge_Stolfi]

This is a sort of process that lead me to this kind of model, that I don't think it might be the real thing, ... I am not telling that it is a random generation or gibberish.

Fine, but (as I noted in my Voynich Day talk) it is impossible to do "agnostic" analysis as you think you are doing.  Any analysis you do will imply certain assumptions about the nature and creation of the text.

In your case, you are implicitly assuming that the text was generated by some uniform mechanical process, hence that it is "gibberish" -- the sequence of letters does not encode some message that the Author wanted to express, but is created autonomously by the process.  And you are also assuming that there are relatively few errors. And that the author himself set the pen to the vellum.  Because you hope that the kind of analysis you are doing will give significant information about that process -- like the mutation rate and mutation process, when the copying resets, etc.

But now suppose that instead the text is indeed a herbal (serious or fantastic) in some natural language, with the spelling or encryption being basically one-to-one on the words.  That is, each word of that ur-language is always written as the same Voynichese word.

Then the type of analysis you are doing obviously makes no sense.  Its results, whatever they are, will not help decipher the manuscript.   Each parag will be an independent unit, and the order of the parags will be essentially "random".   It would make no sense to collect statistics about tokens that are d tokens apart, if they are in different parags.  

In fact, I expect that any statistic will be unchanged if the parags are randomly scrambled, keeping each parag as a rigid unit. If some statistic is affected by this scrambling,that fact will not give any useful clue about the text; we will never understand why that happens until we decipher the manuscript in some other way.

Moreover, each parag will consist of sentences that have a non-trivial heterogeneous structure.  Check You are not allowed to view links. Register or Login to view..  Say, first there will be the name(s) of the plant, then a description of it, then some conditions that it cures or benefits that it provides, then the mode of preparation, then dosage, etc.  Each of these sentences will have its own vocabulary, word frequencies, word order, etc. And each of these will have highly variable length, and may be omitted or duplicated in some parags,  Therefore, even if you limit the statistical analysis to tokens within the same parag, the counts for token pairs that are d tokens apart may be jumbling information about all those sentence types.

As another point, the use of Levenstein or edit distance in the statistics implies the assumption that any character may be replaced, deleted, inserted, or transposed with equal probability, independent of the character or of its position within the word.  That implicitly assumes that the variations are due to certain reasons (like intentional mutation, or errors in the cipher computation) but not to others (like grammatical inflection, sandhi, similar sounds, or similarity of shapes).  Which in turn implies certain assumptions about how the manuscript was created.

All this to show that any statistics that you choose to collect will tacitly imply some "Origin Theory".  

Thus you had better be conscious of those assumptions, and decide whether they are likely or not, before you spend any time collecting tons of statistics that may be useless.

Word choice is constrained by position in the line or paragraph

The dependency on position in the paragraph is quite expected if the text is indeed a herbal, as discussed above.  

As was pointed out many times recently, the dependency on position in a line may well be an artifact of the basic line-breaking algorithms.  Namely, the first word after a line break tends to be longer than average, while the 1-3 words  before a line break tend to be shorter than average. 

This pattern has been verified in texts in any language, independently of the nature of the text and of the distance between the "rails" (edges of the text area).  If the token length distribution is different at those places, the word frequencies will be different too, and then the character and digraph frequencies will be different too.  

Is this phenomenon enough to explain the "LAAFU" effects seen in the VMS?  I don't know, but that is a question that is worth investigating.   One way would be to join all the lines of each parag (excluding the first 1.5 lines)  into a single running text, then break it into lines with different rail spacing.  Say, if the original line has N glyphs on average, then re-format the paragraph into lines with at most c*N glyphs, where c = 1.618 or 0.618, and see if those claimed "LAAFU anomalies" are still observed. (Recognize the number?  Can you tell why that is a good choice for c?).

More generally, collecting statistics with no defined goal is very likely to be a waste of time.  Seriously consider following the scientific method instead: formulate a hypothesis about the nature of the text (like "it is a herbal with one-to-one word encoding", or "it is gibberish generated by a fixed copy-and-mutate style algorithm") , then try to devise the simplest analysis that is likely to clearly disprove that hypothesis, if it is false (rather than a test that is likely to succeed it if it is true).

All the best, --stolfi