05-04-2026, 08:39 PM
I have been doing some testing with different models.
The experiment explores how far simple generative rules can go in reproducing measurable properties of the Voynich text. Instead of focusing on a single metric, several constraints were evaluated together, including repetition rate, local similarity within a fixed window, entropy, hapax proportion, vocabulary size, and positional differences within lines.
Several model families were tested. The simplest ones rely on global sampling, with or without small random variations. More structured versions introduce a local pool of tokens, dependence on the previous word, and eventually a bias depending on the position within the line. The most complex variant also allows a small amount of controlled novelty, generating slightly new forms rather than only reusing existing ones.
The results show that basic properties such as repetition and short-range similarity are relatively easy to reproduce. Many different models can be tuned to reach similar values. The situation changes when additional constraints are included. Differences between models become clearer, and none of them manages to satisfy all constraints at once.
The models that perform best are those combining local reuse, dependence on the previous token, and some positional sensitivity. Adding a small amount of new word formation improves the fit slightly. However, even these models still deviate significantly in several key aspects, especially in hapax rate, vocabulary size, and how words behave at the beginning and end of lines.
The main conclusion is limited but somehow clear: simple local mechanisms are sufficient to reproduce part of the observable structure, but they fail when multiple constraints are considered simultaneously. The difficulty lies not in matching individual statistics, but in matching all of them at once.
The experiment explores how far simple generative rules can go in reproducing measurable properties of the Voynich text. Instead of focusing on a single metric, several constraints were evaluated together, including repetition rate, local similarity within a fixed window, entropy, hapax proportion, vocabulary size, and positional differences within lines.
Several model families were tested. The simplest ones rely on global sampling, with or without small random variations. More structured versions introduce a local pool of tokens, dependence on the previous word, and eventually a bias depending on the position within the line. The most complex variant also allows a small amount of controlled novelty, generating slightly new forms rather than only reusing existing ones.
| Model | Core idea | What it gets roughly right | Main failures |
|---|---|---|---|
| base | Samples tokens from global vocabulary with weak repetition penalty | Basic entropy levels, approximate repetition | Too little structure, wrong similarity, poor hapax and vocabulary control, no positional behavior |
| base_variant | Global sampling + occasional independent small edits | Repetition and some local similarity | Variation not tied to context, wrong hapax rate, no positional asymmetry, weak vocabulary structure |
| prev_morph | New token often derived from previous token (local edit process) | Short-range similarity and chaining effects | Over-relies on sequential edits, fails on repetition balance, hapax and vocabulary distribution |
| positional_prev_morph | Local pool + dependence on previous token + bias by position in line | Repetition, similarity, partial positional effects | Still poor hapax rate, vocabulary size, and incorrect start/end of line behavior |
| positional_prev_morph_new | Same as above + small probability of generating new forms (controlled novelty) | Best overall balance across repetition, similarity and entropy | Large errors remain in hapax rate, vocabulary, word-length asymmetry and positional distributions |
The results show that basic properties such as repetition and short-range similarity are relatively easy to reproduce. Many different models can be tuned to reach similar values. The situation changes when additional constraints are included. Differences between models become clearer, and none of them manages to satisfy all constraints at once.
The models that perform best are those combining local reuse, dependence on the previous token, and some positional sensitivity. Adding a small amount of new word formation improves the fit slightly. However, even these models still deviate significantly in several key aspects, especially in hapax rate, vocabulary size, and how words behave at the beginning and end of lines.
The main conclusion is limited but somehow clear: simple local mechanisms are sufficient to reproduce part of the observable structure, but they fail when multiple constraints are considered simultaneously. The difficulty lies not in matching individual statistics, but in matching all of them at once.