In simple terms this means that in most natural languages if you takes some of the most common tokens (for example, "and", "the", "ing") they won't normally appear in texts in all kinds of pairwise combinations. There will be no or little "the and" or "and ing" or "the ing", even though the reverse combinations would be frequent. There would be little of "the the", "ing ing". According to the charts, this is observed in English, Latin, Arabic, Chinese expressed as characters, Chinese expressed as pinyin, English with spaces removed. However, this is not observed in Voynichese, spaces or no spaces. If there is some very popular token like daiin or chol, there would be some chol.daiin and there would be some daiin.chol in the text.
The only difference between this example and the real algorithm is that the text is split into tokens using a variation of BPE algorithm and not relying on any predefined boundaries (like spaces or anything else).
The only difference between this example and the real algorithm is that the text is split into tokens using a variation of BPE algorithm and not relying on any predefined boundaries (like spaces or anything else).
![[+]](https://www.voynich.ninja/images/collapse_collapsed.png)