How multi-character substitution might explain the voynich’s strange entropy

[oshfdk]

The unscrambled Cuva sample (continuous text) is shown as grey stars, in satisfactory agreement with your results.  They show that the VMS generator is correlated on a scale of 2-4 characters, before the supposed-entropy curve is degraded by its underlying numerical limitations.  Any features appearing farther to the right are suspect.  We already know that sample length shapes such curves;  to make a strong case that encoding method also has a differential effect on them, some kind of sample-length validations would help.  How large is your Docta Ignorantia?  and how is its length affected by the transformations applied?

Thank you for the explanation, I was a bit struggling to follow the discussion otherwise.

As far as I remember, quimqu was experimenting with GPT models. I think it still could be useful to try longer n-grams with a small GPT model, if the size of the model won't let it memorize all patterns of length n. If it turns out a model can improve its score (get lower perplexity) for high order ngrams despite not having enough weights, it would mean there are some rules or correlations between characters separated by n positions, and the model is able to learn these rules. Does this sound plausible?