TTR Analysis, Observed vs. Reference Curve

[Jorge_Stolfi]

I noticed that TTR (and the exponent of Heaps) decrease with N. The larger the section, the smaller the expected TTR.

Heaps's law says that the wordtypes-to-tokens ratio (TTR) decreases as the number of tokens N increases. However the exponent r that determines how fast that happens is supposed to be constant for all N, even as low as thousand or so.  

Thus there is no point in "separating size from behavior": the behavior is how the TTR changes with size.

And that is what I see in @bi3mw's plots.  There are two sections that fit Heaps's law quite well: Herbal and Balneo (Bio).  The TTR plots follows the theoretical curves quite well overall, and almost exactly starting at N=3000 tokens or so.  

In fact, the small discrepancies below 3000 and above 8000 seem to be due to bad fitting of the theoretical curve, rather than "non-Heapsiness" of the text.  I bet that the discrepancies would practically vanish if the curve-fitting procedure was given only the data for N > 500 (or if each data point was given a weight proportional to N).  The plots should still use the whole range of N, of course.

On the other hand, the large deviations of the other sections mean that those texts just do not follow Heaps's law.   Then there is no much point in trying to get them to follow the law for large N.  Astro, Cosmo, and Pharma are too small for that anyway.  As for Recipes, one could try to fit the law to the range 6000-11000, but I don't know what conclusion one could get from the result.

However, an exercise along the lines you propose may be interesting.  One could just run the test for the second half of the Recipes section.  I suspect that the second half will be found to follow Heaps's law more closely than the first half.

(Somewhere in Recipes there seems to be an abrupt change of handwriting in the middle of a paragraph, or even in the middle of a word, a few lines after the anomalous right-justified mid-page "title".  I like to imagine the Author firing the Scribe for that and other blunders, and hiring another one. Then the change in the Heaps plot behavior may be due to the first Scribe making lots of errors that depressed the TTR.)

One could also try to randomly scramble the words of each section.  I expect that it will not make much difference for Herbal and Balneo, but it should smooth out the plot of the other sections, maybe even make them fit the Law.

All the best, --stolfi