Why the Voynich Manuscript Might Not Be a Real Language

[quimqu]

Ah, I see. So you have a sliding window of n, and then you see how well you can predict the next n characters? I just can't intuitively grasp what this tells us. It's too abstract - a measure of the information density of the writing system. 

Actually, maybe it was me who misunderstood what quimqu was computing. According to the intro post: 

An n-gram model learns the likelihood of seeing a particular character given the previous n-1 characters.

So, for 5-gram it should be guessing the next character based on the previous 4 characters.

For example, given qoke*, what is *?

But then it's strange that the perplexity goes up with longer ngrams, it shouldn't.

So, I don't understand what is going on here.

Yes, your understanding is correct now: for an n-gram model, the prediction is always made based on the previous n–1 characters. So a 5-gram model estimates the probability of a character given the 4 characters before it — exactly like your example:

given qoke*, what is *?

However, there is a subtlety in how perplexity is calculated, which may explain the unexpected behavior you're seeing — especially the fact that perplexity increases for higher n. That seems counterintuitive, because more context should normally make predictions more accurate (and perplexity lower), right?
But here's why that’s not always the case in practice:

Why perplexity can increase with higher n

1. Data sparsity
   As n increases, the number of possible (n–1)-length contexts grows exponentially. Most of these will never appear in the training data, so the model encounters many "unknown" contexts at test time. That leads to fallback probabilities (e.g. uniform guesses or smoothed estimates), which are typically worse than the lower-n generalizations.
   
2. Voynich text characteristics
   In natural languages, character sequences are often quite predictable due to morphological and phonotactic constraints. But in the Voynich manuscript, the information seems more evenly spread across the word. That means longer contexts don’t always help more than shorter ones — they just become more brittle.
   

So yes — in theory, a 5-gram model should outperform a 3-gram model. But in practice:

• If the data is sparse
  
• If the structure is not very redundant (as may be the case in Voynichese)
  

Then longer n-grams may struggle due to overfitting or lack of seen contexts, leading to higher perplexity.