Repetition of words

[Jorge_Stolfi]

The VMS (*) has 38411 words (**) in total, and 8424 unique words (***), that is to say 1 word type every ~4.56 word tokens. This is just a slightly higher percentage than for instance Caesar's De Bello Gallico (1 word type every ~4.67 word tokens).

If a language follows Zipf's law, the token/lexeme ratio in a sample cannot be a constant.  As the number N of tokens (word occurrences) in a sample increases, the number M of lexemes (distinct words) grows like K*sqrt(N).  More precisely like K*N**b where b is typically between 0.4 and 0.6.  This formula is known as You are not allowed to view links. Register or Login to view..

So, when comparing the VMS lexicon size to that of other languages, it is important to use samples with the same number of tokens.  

Assuming the exponent b is 0.5 for both languages, the interesting language parameter (independent of  sample size) is K = M/sqrt(N), not M/N.

All the best, --stolfi

All the best