15-09-2019, 04:37 AM
Pliny's text is an interesting model text, and perhaps not 'typical'.
By its encyclopedic nature, it has more word types (different words) than the average text.
The h0 values shows that at 70,000 words it has 21,936 word types. I don't think that many texts will even come near this. Effectively, h0 and h1 grow faster than would be typical.
Also, I've come to the conclusion that the h2 that I estimated before, namely:
2log (Ntoken) - h1
represents a theoretical maximum.
For the Pliny samples, the actual h2 is very close to this maximum.
At 1000 words it is 97% of the maximum and at 70,000 words it is 91%.
For this text, the h1-h2 is relatively stable. Its value is higher than I originally expected, and this is clearly due to the Zipf law model that I applied, which doesn't fit this text, and perhaps also not many texts.
By its encyclopedic nature, it has more word types (different words) than the average text.
The h0 values shows that at 70,000 words it has 21,936 word types. I don't think that many texts will even come near this. Effectively, h0 and h1 grow faster than would be typical.
Also, I've come to the conclusion that the h2 that I estimated before, namely:
2log (Ntoken) - h1
represents a theoretical maximum.
For the Pliny samples, the actual h2 is very close to this maximum.
At 1000 words it is 97% of the maximum and at 70,000 words it is 91%.
For this text, the h1-h2 is relatively stable. Its value is higher than I originally expected, and this is clearly due to the Zipf law model that I applied, which doesn't fit this text, and perhaps also not many texts.