Cut out - LLM compared to random

[Jorge_Stolfi]

- cut out random 10 pages off the full text (voynich)
- train a base llm with the text (10 pages missing).
- give one of the cut out pages to the llm with one EVA-Token missing. the llm guesses a token for the missing one.
- repeat this 20 times with different tokens missing.
- do the same with a random text
- do the same with a generated text (from one of supposed solutions for example).
- compare the probability.

Using an LLM to predict a missing word does not seem very different than using a second-order Markov model (that predicts the next word W3 from the last two words W1,W2, based on the frequencies of word triples W1 W2 W3 measured on the VMS itself.  Or any variant thereof, e.g. predicting W2 given W1 and W3.

One difference is that maybe the LLM uses a more adaptive strategy, using more of fewer words of context depending on the local situation.

Another difference is that you will have no idea of what the LLM is actually doing.

In general, higher-order Markov models will get better and better at reproducing the training text (the subset of the text from which word tuple statistics are collected).  At some point they will essentially memorize the training text, because (say) each triple W1 W2 W3 occurs only once in it, and thus the model will have only one choice for the next word W4.   Then the model will get very good at predicting it -- and very bad at predicting the other half of the text, omitted from the training.

If the lexicon has N words, and all of them occur with roughly equal frequency, the training text for an order-k Markov must have a few times N^(k+1) words; so that the frequency of W[k+1] given W1 W2...Wk can be measured from it, for all (k+1)-tuples.   Namely, if the lexicon has 2000 words, you need a text that is tens of millions of words in order to properly train even a first-order Markov.  

It is much worse if the words have highly unequal frequencies, as predicted by Zipf's law.

Said another way, an order-k Markov has N^(k+1) internal parameters, so you need that magnitude of training data to even begin adjusting them to the proper values.

LLMs have this same problem.  Depending on how many internal parameters the LLM has, you may need a training text with billions of words in order for them to get remotely close to proper values.

Then there is the question of what is the "random" text you are using as control.  For instance, if the "random" text is generated by a second-order Markov, a predictor that also uses a second-order Markov may have the same success rate on both texts, I don't know...

All the best, --stolfi