How LLM models try to understand Voynichese

[oshfdk]

The model tends to predict the most frequent words in the Voynich corpus, as expected. However, the saliency values let us observe which previous words influenced the prediction the most, token by token.

I highlighted:
• green: when pred_word == target
• yellow: when more than 50% of the characters in pred_word match the target (partial match)

Am I reading this correctly? It looks like GPT cannot successfully predict the fifth word knowing the previous 4 words. Correct predictions seem very rare and are the most common words like "daiin" or "chol", so this looks like pure chance to me.

If so, I'm not sure of how much use the saliency is when the predictions themselves are wrong. For a model that successfully predicts a large percentage of words (without overfitting) it would be very interesting to know which information it uses, but not for a model that cannot predict much.