(04-11-2019, 08:16 PM)Torsten wrote: You are not allowed to view links. Register or Login to view.Keep in mind that the VMS token strings fulfill both of Zipf’s laws. This fact is frequently seen as important evidence for the presence of human language and that vords are words (see You are not allowed to view links. Register or Login to view.). A different mapping would also result in changed statistics. Therefore the mapping should preserve the word frequencies. Otherwise a different explanation for the observed word frequencies would be necessary.
Word frequency is a matter of opinion... see for example the huge difference in MATTR that Koen computed using the two versions of the ZL transliteration, the one with less spaces and the one with more spaces. Getting "evidence" out of unreliable data is unconvincing.
Let's stay on topic: how to choose between all the imaginable causes of local self-similarity? I don't know.

