Heap's Law

[davidjackson]

MarcoP, it's always a pleasure to be corrected by you. Possibly because I have no academic reputation to maintain, it's easier for me to do something foolish and laugh it off, but I do my little best to be rigorous.   But this is what the forum is for, to advance knowledge by bouncing ideas off one another - not to present pre-packaged tidy little snippets!

Anyway, back to the thread.
I originally became interested in Heap's Law thanks to the You are not allowed to view links. Register or Login to view. - it seems a good way of quickly proving whether a generation method could be authentic or not.

I have run a sample against the text generated by T.Timm and found You are not allowed to view links. Register or Login to view.. The graphed result is this:
   
(NB: Caption "total words in texts" should read "unique word count")

If we compare this to the result found in the stars section, here:
   

We see that Timm's text is far more regular in its creation of new words than the stars section is. I have added a trend line (green) to both graphs to more clearly illustrate this.

Both samples have a similar sized corpus. Timm's consists of 10,210 words, the star section 12,974. I chose the star section as it is almost illustration free.

A further difference is in the predicted rate of unique word creation. The stars diagram above has an average difference between real and predicted creation of unique words of just 13. Timm's average difference is 114.

If I change the formula to read B=0.56 for Timm's, then I get an average difference of -16. But the lines do not fit as well.

Since the ratio of creation is constant, I get two rapidly diverging lines, as illustrated when I overlay two trend lines and expand them outwards:
   

This graph shows the trend lines expanded outwards for the stars section. Note that the real creation line is variable, but tends to follow the Heap's Law prediction, unlike its trend line.
   

To an extent, the Timm's result could be predicted. It is computer generated and as such is following a constant generation algorithm.

The human generated text of the VMS is not following a constant algorithm. Instead, it appears to be creating an inconstant flow of new unique words, which may -and only may- be linked with the introduction of new topics within the text. Alternatively, this could be down to the scribe falling into a pattern when creating random text and repeating previous work.

I attach the two spreadsheets used to generate the graphs for reference.