The structure of the Voynich text and how it may be generated

[quimqu]

I know that my posts are quite dense and "hard" to read. So I try to summarize here:

What remains consistent with the data is not a chain of transformations, but a constrained selection process. The line defines a space of valid forms, the local context restricts this space further, and the final choice is made among multiple equally compatible candidates. This shifts the problem from “how is the next token generated” to “how is the space of valid tokens defined”.

The one can think of concrete and testable (I will work on them) assumptions about the mechanism. Instead of trying to predict a single next token, the model should first approximate a candidate set, and only then model selection within that set.

Assumption	Operational test
Candidate set constrained by line position	Predict top-k tokens from line features and measure recall
Local context restricts form, not identity	Build candidates by similarity and check if real token is included
Selection within the set is weak	Train ranking model and measure score flatness
Substructures drive compatibility	Use prefixes/suffixes only and test predictive power

With this, a generative model would not sample tokens directly from a sequence model, but proceed in two steps: first generate or retrieve a set of compatible forms given the line and its context, then sample from this set with relatively low discrimination. The key quantity to model is no longer the probability of a token given the previous one, but the probability that a token belongs to the current feasible set.

This leads to a different type of validation. A good model is not one that predicts the exact next token, but one that consistently places the real token inside a small, well-defined candidate set.

I hope to have explained myself better