GPT Models Fail to Find Language Structure in the Voynich Manuscript

[quimqu]

It would be interesting to see what your network would do if you mapped every a to o, every s to r, every ch to ee...

Thanks, Jorge

lots of great points!

I agree that with such a small corpus, the model mostly memorizes sequences, especially at low context lengths.

In fact, I trained several models with different block sizes (from 1 to 32). Block size tells how many tokens are used for learning and predicting. As expected, with block size 1 the results are better — the model just echoes bigrams. But with larger blocks (e.g., 8 or more), it needs to rely on broader context, and that’s where Voynichese starts to diverge from natural languages.

Here’s a sample of matching bigrams and trigrams for block size = 8, for several texts including Voynichese:

You are not allowed to view links. Register or Login to view.

Format is so: each ngram is marked in green if it appears in the original book. Files are in html, so I suggest to open them in Google Docs.



For example, the first line of the image: 

- the GPT output was: sharo.ydy.cheal.omororom.chol.pcheo.olkaim
- in the htm file, you will see bigram by bigram (or trigram by trigram), this is sharo.ydy ydy.cheal cheal.omororom omororom.chol chol.pcheo pcheo.olkaim
- these bigrams are compared against the original corpus and highlighted in green if they appear in the exact same order, or in red if they do not (i.e., the two words never occur together in that order in the original text).
- the GPT does not create new words, all words are valid ones. The creation of new words is studied in another thread, where I trained GPT character by character.

Also, I’ll definitely try the substitution experiment you suggested — very curious to see how that affects predictability. Thanks again