A key to understand the VMS

[Sam G]

For me the text generation method was coming obvious while looking on rare words. See for instance the word "lkl". This word exists only 9 times for the VMS. But this doesn't mean that this word is equally distributed. In fact there is one page with three instances of "lkl". This page is You are not allowed to view links. Register or Login to view. and since three instances of "lkl" can be found on this page it was three times available as source for generating other words. This is the reason that on page You are not allowed to view links. Register or Login to view. also the words "lklor", "kl" and "lkol" exists. Beside "lkl" only 6 other words contain the sequence "lkl". One of this six words is the word "lklor" on page f105v. This is not a coincidence! See "lkl" / "talkl" on page You are not allowed to view links. Register or Login to view. and "lklcheol" / "lkl" on page f115v.

This seems to be a normal property of meaningful texts.  Here's a post by Jorge Stolfi where he looks for patterns like this in War of the Worlds:

You are not allowed to view links. Register or Login to view.

Along the same vein, these words occur exactly twice, at most 20
tokens apart:

VMS:

   FIRST    LAST    SPAN WORD
   -----   -----   ----- ---------
     644     650       6 damo
    6713    6731      18 olchdaiin
   16965   16969       4 cheteey
   27756   27757       1 qoekol
   36106   36116      10 otaraiin
   36462   36465       3 chtl

WotW:

   FIRST    LAST    SPAN WORD
   -----   -----   ----- ---------
     789     791       2 generation
    5156    5159       3 a~screwin'
    6428    6430       2 joint
    6531    6538       7 ugly
    6532    6539       7 brutes
    6914    6915       1 flutter
    8207    8214       7 novelty
   11097   11104       7 fringe
   15267   15268       1 aloo
   17366   17370       4 shield
   18391   18403      12 streamed
   18414   18422       8 pillars
   23663   23665       2 rows
   29884   29897      13 uppermost
   31994   31997       3 losing
   33642   33655      13 deepened
   34090   34094       4 girls
   35385   35399      14 ellen
   35568   35571       3 garrick
   36993   36997       4 stampede
   37729   37738       9 midland

Note that these include the successive repeats "qoekol qoekol",
"flutter flutter" and "aloo aloo".

As before, the scrambled files had no hits --- i.e. all words with two
occurrences were spread wider than 20 tokens.

In conclusion, both books contain many words that are confined
to specific sections, far more than expected by chance.
Of course this does not prove anything, but is yet another
constraint on proposed theories.

Surprisingly, the WotW has more "lumpy" words than the VMS. This may
be due to the fact that different sections are partly mixed in the
VMS.  I need to find a metric of "lumpiness" that degrades more
graciously with such block-scrambling.