GPT Models Fail to Find Language Structure in the Voynich Manuscript

[magnesium]

For the results to be comparable, we need to establish how Voynichese words correspond to plaintext words (and if they are words and not syllables, character combinations, etc). I think it makes sense repeating the experiment using comparable training sizes and output sizes expressed in Shannon bits

I see your point, but I wonder if that distinction might be overemphasized. Whether the tokens represent words, syllables, or even phonemes, any genuine writing system that encodes natural language should still reflect the underlying linguistic structure — which GPT models are designed to pick up on, regardless of granularity.

For example, Chinese uses characters that are often closer to morphemes or syllables than words, and Latin is highly inflected with longer words — but both still have strong statistical patterns in their sequential structure. If Voynichese encoded a real language at any level — word, syllable, or otherwise — we’d expect some degree of learnable co-occurrence patterns.

So while it’s an open question what exactly Voynichese tokens represent, the GPT’s failure to learn even short sequences suggests something deeper than just a mismatch in segmentation level. It might be structural.

This could be interesting to test systematically with different kinds of ciphertexts. Are there classes of substitution ciphers, for instance, that produce readily decipherable ciphertexts that exhibit anomalously low, VMS-like GPT predictability? If a given type of cipher, encrypting a wide range of plaintexts, consistently appears to be more GPT-predictable than the VMS, than that kind of cipher probably isn't consistent with the VMS. But this method probably would be overkill: Monoalphabetic substitution ciphers would probably show up as much more predictable in this analysis than the VMS, but we can rule out monoalphabetic substitution ciphers using much less computationally expensive techniques.