Vord frequency histogram as an indicator of the text category

[ReneZ]

While the Zipf law model is far from perfect, it gives at least a first approximation we can work with.
One aspect that seems not to be mentioned a lot is that the inverse frequency law does not really work for the highest frequency words, i.e. the line with slope -1 on a log-log scale tends to flatten a bit at the left.
However, we can leave that out of consideration for the moment.

If the text had 37,919 words (figure from Reddy and Knight), we can predict the frequency table for all word types. The predicted number can be obtained by rounding.
In this case, the cut-off is when the expected frequency of a word is less than 0.5.

We then find the following numbers:
Number of different words (word types): 7738

This is a bit higher than what Anton finds, but I could redo the stats with the number of word tokens in the Takeshi transcription, if he has that number. In any case, the frequency curve is very flat in this area, so the actual cut-off value is quite ill-defined.

Number of words that occur once: 5159
This is 67%, spot on.

101-200 times: 19
201-300 times:  7
301-400 times: 3
More: 9
Total over 100: 38

These numbers are actually lower than in the MS.