VM TTR values

[Koen G]

I noticed that when shuffling, I often obtain a similar shape:

   

The shuffled versions are relatively low on the left. This is because of the phenomenon Marco described before: in natural language we won't write "the the", but there are so many of them in the text that random shuffling will put some of them next to each other. So in this case, natural language "avoids" some of the reduplication that would normally occur statistically. 

Then there is a sharp rise between 5-30, something like that. This is the area where natural languages will apparently do things that decrease TTR, so shuffling undoes this.

Finally, the line descends again towards the higher windows, because here shuffling vs not shuffling doesn't make much of a difference. If you take a window of a 1000 words, its vocabulary composition will be similar whether the text has been shuffled first or not.

Now the thing is, irrelevant of the initial shape, graphs for shuffled texts always seem to do the same: start low, then climb quickly, then slope down. This is a result of the mean and stdev I use for normalization, which is based on unshuffled texts. Maybe if I use mean and stdev values based on shuffled text, the graphs might be more informative. I need to think about this some more.