02-03-2016, 10:07 AM
As an example, the VM contains 33319 distinct four-word sequences when paragraph boundaries are respected. If characters d and e are discarded, that number goes down to 33263, a difference of only 56.
Other character pairs yield a higher number of distinct four-word sequences, but not by a significant amount, so d and e are not really special in this regard.
As we remove additional characters, an increasing number of words begin to disappear and the results become less meaningful. Empty words were treated as non-empty to enable comparison with the original number of four-word sequences. Otherwise, discarding d and y is slightly more effective than discarding d and e given that dy occurs as a word 271 times.
It's not particularly revealing and not exactly an in-depth analysis, but it suggests that the low number of repeated four-word sequences is not simply a result of random insertions of one or two null characters.
Other character pairs yield a higher number of distinct four-word sequences, but not by a significant amount, so d and e are not really special in this regard.
As we remove additional characters, an increasing number of words begin to disappear and the results become less meaningful. Empty words were treated as non-empty to enable comparison with the original number of four-word sequences. Otherwise, discarding d and y is slightly more effective than discarding d and e given that dy occurs as a word 271 times.
It's not particularly revealing and not exactly an in-depth analysis, but it suggests that the low number of repeated four-word sequences is not simply a result of random insertions of one or two null characters.