About word length distribution

[kckluge]

Ah, I understand.

I don't know nearly as much about statistics as most of the other guys here, but isn't Zipf's law specifically something that shows up even when no apparent "Zippifying" method lies at the basis of the data?

Koen,

I'd phrase it somewhat differently as there are processes that will generate pseudo-text whose vocabulary will show a Zipf's Law like frequency distribution, but yes. And per tikonen's comment You are not allowed to view links. Register or Login to view. (and as Rugg pointed out regarding his grill method) there are ways of generating pseudo-text whose vocabulary has a binomial length distribution.

As for your earlier question in the thread "why would they go through this effort?" -- without drilling down into details (because if I had gotten my preliminary experiments to a publishable point I would have written them up more formally), in the context of verbose cipher theories:

* an obvious issue with a verbose cipher (and an obvious tell) is that the enciphered words are longer than the corresponding plaintext words;

* while a mechanism like rule-based insertion of spaces (i.e., always before EVA 'q' or after EVA "ain"/"aiin"/etc sequences) helps, it doesn't help enough;

* so let's cut the Gordian knot, and assume some probabilistic process like the sum of multiple dice or coin flips is used to break the text up;

* in that context, the potted history of frequency analysis you'll find in many textbooks is wrong -- frequency analysis of dice throws goes back at least as far as Dante (so plausible for early 15th century);

* combining a) a verbose cipher using the obvious verbose Voynich glyph combinations as some of the elements mapped to (say) Latin based on relative frequency rankings with b) a sum-of-(dice/coin flips) based method for breaking the text up shows promise for potentially replicating all of (1) a binomial-like vocabulary word length distribution, (2) a Voynich-like type/token ratio and hapax fraction, and (3) a vocabulary showing Zipfian frequency behavior with a log-log slope fit similar to the Voynich text.

In addition to some modifications to what I was doing in my preliminary experiments to try to improve the results, whether the text generates the type of distribution of distance between similar words seen in the Voynich text is an open question.