Bit-level Geometry: A deterministic discovery of Voynichese building blocks

[nablator]

If we find a pattern that repeats, it means the author was following a rule. If the engine shows that sh always connects to edy, that is no longer a coincidence - it is a law of the language.

Hi,

The problematic thing is: there is no law, only local preferences. There have been many attempts to tokenize the text into smaller chunks that could be used as building blocks, all of them unconvincing IMHO. The idea is to increase the character conditional entropy, not only (Shannon's) character entropy.

The math proves exactly where the author(s) of the manuscript connected their building blocks.

I don't think it does. Different compression algorithms produce different results and there is no guarantee that the optimal compression results in the best tokenization or that tokenization is even necessary to parse the text.

For example the tokenization used by LLMs (see You are not allowed to view links. Register or Login to view.) is not the one that maximizes compression. Best related to meaning is not the same as best compressed.

There is a dictionary in every zip file (the LZW algorithm uses a dictionary). If you zip a text file in English, does the content of the dictionary reveal some deep truth about the building blocks of the English language? Of course not. It may identify some useful morphemes (prefixes, roots, suffixes) but not all, it only identifies frequent repetitions and not how the language is put together.