Bit-level Geometry: A deterministic discovery of Voynichese building blocks

[TheEnglishKiwi]

That claim is questionable already at the theoretical level

Hi Jorge, a huge thanks for taking the time to give some excellent feedback! Allow me to provide some critical clarifications as I believe you are looking at the tool through the lens of traditional, generative linguistics. The tool is strictly an extractive, descriptive compression engine. 

Important distinction between "Generative" and "Descriptive":
You are absolutely right, language is not a disjoint union. However, the engine isn't trying to write a rulebook for how to generate new Voynichese words (where L = A x B falls apart due to exceptions).

MDL doesn't care about generating new words, it is strictly descriptive. That means it treats the entire manuscript as a closed universe of data. It doesn't attempt to claim that any prefix can attach to any suffix, rather - it simply draws a mathematical map of the pieces that already exist on the page. It simply states "In this specific manuscript, Piece A connects to Piece B exactly X times".

Phonetic Variations:
You correctly pointed out that roots change spellings when suffixes are added. But MDL doesn't rely on perfect/uniform phonetic rules. If the text uses "-able" and "-ible", the engine simply identifies them as 2 separate, highly efficient suffix blocks. It maps the messiness, exactly how it is, without ever needing to understand the morphological rules behind it.

Global Disctionary Optimisation vs Semantics
In regard to the assumption that you need to know the meaning of a word in order to know where to cut it, MDL solves this through Global Dictionary Optimisation. Let's compare 2 scenarios using the "unbelievable" example:

If the engine cuts "unb-", it has to create a new entry for "unb-", which can only be used a few times in total (e.g. "unbelievable" or "unbalanced"). The mathematical cost of storing "unb-" far outweighs the compression space saved.

If the engine cuts "un-", it can reuse that exact block 100's of times across the entire text (e.g. "unhappy", "undo", "unseen", "unknown", "unusual" and so on). It mathematically proves that "un-" is the true structural block because it returns the highest global compression ratio, all without needing to know what the words actually mean.

Authors Intent:
As I replied to another comment, I do need to concede this point. You are 100% correct there is no proof of psychological intent from the author, rather simply using math to prove the structural rules of the text. The math definitely proves the structural rules of the text to show how it was physically assembled, but not what was in the author's head.

Given your own extensive work on Voynichese word paradigms, are there specific 'rules' you've observed in Voynichese that you believe a pure compression algorithm would blindly fail to capture?

Cheers,
Ben