Bit-level Geometry: A deterministic discovery of Voynichese building blocks

[TheEnglishKiwi]

I don't think it does. Different compression algorithms produce different results and there is no guarantee that the optimal compression results in the best tokenization or that tokenization is even necessary to parse the text.

For example the tokenization used by LLMs (see You are not allowed to view links. Register or Login to view.) is not the one that maximizes compression. Best related to meaning is not the same as best compressed.

There is a dictionary in every zip file (the LZW algorithm uses a dictionary). If you zip a text file in English, does the content of the dictionary reveal some deep truth about the building blocks of the English language? Of course not. It may identify some useful morphemes (prefixes, roots, suffixes) but not all, it only identifies frequent repetitions and not how the language is put together.

Hi there,

Thank you for excellent feedback, I appreciate it!

1 (LZW): So the issue with LZW is it's a dumb, greedy form of compression. It'll read text left-to-right and grab whatever repeats first to save space quickly. Our engine uses Minimum Description Length (MDL). We evaluate the mathematical cost of the entire dictionary which penalises dumb chunks that LZW would make. MDL is considered the gold standard for finding roots and affixes without knowing the language (See: John Goldsmith).

2 (LLM/BPE): LLM's are terrible at finding linguistic roots because they use tokenisers like BPE which merge the most frequent pairs of bytes together and this results in what you are saying which is: "Best Compressed" does not equal "Best Meaning". We use geometric centroids and evaluate the global boundary tax, not just the raw frequency.

3 (Conditional Entropy):  I understand you saying that we need to look at conditional entropy, not just Shannon entropy, but that's actually describing exactly how our engine works. When a sequence has an extremely low conditional entropy (e.g. "sh" strictly predicts "edy"), an MDL engine groups them together into a single block. By doing so, the entropy between the newly formed blocks go up.

I must admit however, I was too enthusiastic to say "it is the law of the language". The math maps out the physical structure of the text, not the underlying semantics.

Thanks!