How LLM models try to understand Voynichese

[Jorge_Stolfi]

Am I reading this correctly? It looks like GPT cannot successfully predict the fifth word knowing the previous 4 words. Correct predictions seem very rare and are the most common words like "daiin" or "chol", so this looks like pure chance to me.

I You are not allowed to view links. Register or Login to view. a few days ago.  To train any word predictor that uses the last K words, you need a corpus in which all possible sequences of K words have a good chance of appearing.  If the lexicon has1000 equally likely words, the corpus needed for that training must have on the order of 1000 to the power K words.  The number will be smaller if the lexicon words have a Zipf-like frequency distribution; but not much smaller.

If you insist that the predictor must use the last 4 words, and you train it with half of the VMS, you end up with a predictor that just copies long stretches of that training text, with only occasional switches when it jumps from one piece of the training set to another.  Namely, those switches will occur  only when the generated output has K consecutive words that occur more than once in the training set -- which will be very few of them.

If you use a black-box predictor, you never know how many past words it actually uses.  Or what the hell it is doing.

All the best, --jorge