(28-06-2020, 07:08 PM)Anton wrote: You are not allowed to view links. Register or Login to view.I think a random text (where each word is a random string constrained by length) will have normal distribution of word type frequency.
Does anybody have a sample?
I saw now that there are already some done experiments posted in this thread. But true, this will not give a normal distribution. If you treat space as just another character with a constant probability of occurrence s, this will give a probability p(n) where n is the length of a word, equal to
p(n) = s*(1-s)^{n-1}
This is a power-law distribution which is asymmetrical (it has a long tail which is always decreasing). The average word-length will be 1/s -1. If you constrain the length you will only cut off the tail.
Edit: This will be a distribution with respect to all words, not word-types. I.e. shorter words will more likely be identical in a random string, therefore making the distribution word-type will decrease frequency of shorter words in the distribution.


![[+]](https://www.voynich.ninja/images/collapse_collapsed.png)