(05-06-2026, 04:39 PM)Labyrinthinesecurity wrote: You are not allowed to view links. Register or Login to view.interesting,.. which kind of metrics do you have in mind?
The one that Torsten Timm used: average edit distance between all words of different lines. See the comparison with other texts here: You are not allowed to view links. Register or Login to view.
I guess there should be no detectable effect (local inter-line similarity) on your generated text.
I wrote a babble detector very recently based on Byte Pair Encoding, not optimal compression I know, so the results are a bit random, but it does seem to find the babble-like sequences quite efficiently (in a window of n words) so it's good enough for a quick comparison of the amount of such sequences. I suppose your generated text should behave like a word-shuffled version of the section(s) it has been trained on, i.e. less babble-like.
For the comparison to be more significant I guess some LAAFU statistics and inter-word statistics would be helpful, they might add a little bias in local similarity. They probably don't matter for the average edit distance between all words of different lines but might impact a little the babble detector on very short sequences of words. For this I used character trigrams statistics with special characters for line start and line end in my simple Markov chain generator (without word model, so the word length distribution was very wrong).


![[+]](https://www.voynich.ninja/images/collapse_collapsed.png)