The Voynich Ninja

Here's a very quick preliminary test. In my opinion, the result is acceptable for such a short text.

[attachment=15818]

(28-05-2026, 09:58 PM)bi3mw Wrote: You are not allowed to view links. Register or Login to view.Here's a very quick preliminary test. In my opinion, the result is acceptable for such a short text.

And you see what I was saying about word length.  You have no 1, 2 or 3 length words so you distribution is way to the right.  Voynich is word length 5 centered, not 6.

Look at this output from my generator:

Output tokens: 8274
Output vocabulary size: 2525
Output hapax count: 1551
Voynich comparison tokens: 8519
Voynich vocabulary size: 2436
Voynich hapax count: 1642

So, your vocabulary is in the ballpark, maybe a little light.  You add those length 1 and 2 tokens and that'll bring up that vocabulary count and likely bring your word length closer to the Voynich if you keep the token count the same.

Now, look at my top 20 bigrams and trigrams compared to the Voynich.  It still needs some tweaking but it's in the ball park.  Is yours?


Look at my word length compared to the Voynich then compare it to yours.

[attachment=15819]

Look at my zipf curve compared to the Voynich. Does yours compare?  I'm betting that until you fix the word length it'll be a "no".

[attachment=15820]

These are the the very basic numbers a generator needs to be "plausible" with before you can even call it a generator.  AND, to be a workable generator it has to be mostly blind about it.  I do use a seed page. After that, it generates the rest on it's own just using my ledger.  And all the numbers you see mine generate are emergent.  I didn't create any code to force a zipf curve.  I do have code that manages word length and that's something a scribe could do with their eyeballs.

The big thing you have to think about.  If you sat down and tried to create something that looked like the Voynich and YOU were in the 15th century, what "tools" would you need other than a quill, some ink and maybe some ponce to erase things with.  Any generator we come up with today needs to be human 15th century doable. Which is why the Naibbe cypher looks great and would technically be human doable but would require dice and cards and lookup tables... extremely cumbersome.  Those are the type things you need to consider.  

Creating Voynich-like text is easy.

Creating a method that explains how a 15th century scribe could do it, that's the hard part.

Edit: If you want to really see how you're doing, put up some real Voynich comparison charts, word length for word length.

Ran some new tests on my generated code vs the Voynich.

This test measures novelty, predictability, word beginnings, word middles, word endings, and how strongly words cluster into families.  It's from some code I developed months ago.  It basically trains on half the text then tests on the other half to see how they compare.   

In this test the generated code is mostly in the ballpark with the exception of bigram reject. That's basically a measure of, look at the bigrams on one half of the text and then see which of those bigrams are in the other half.  In the case of my generator, it's pretty low which means, it's producing too many close relatives. It stays inside the learned families extremely well, but the real herbal text appears to generate more new combinations while still obeying its structural constraints. 

BPC is bits per character which measures entropy or how predictable text is.  BPC2 uses bigrams, BPC3 trigrams.  ΔBPC is the difference between the two.

The ED1 tests look at how many words in the training set are edit distance 1 or copies of words in the test set.  Token asks how many words have a close relative in the test set and type asks how many unique words have a close relative.  Again, it's off by a bit and is probably related to the bigram reject difference.  97% of the unique training words are ED 1 or 0 to a test word.  

And I think I know why these are issues. I tried to keep everything a copy or ED1 from it's source and it almost never generates a random new word.  It appears there's a bit more novelty in the Voynich than I'm allowing for.

So again, not perfect.  In the neighborhood?  Mostly, but needs work.

Metric	Herbal Voynich	RK42	Difference
Bigram reject %	0.5655	0.1044	much lower
BPC2	2.4472	2.8717	higher
BPC3	2.1978	2.5697	higher
ΔBPC	0.2494	0.3020	higher
Initial BPC	3.0692	3.1479	slightly higher
Medial BPC	3.3306	3.4473	slightly higher
Final BPC	2.6014	2.9747	higher
ED≤1 Token	95.7988%	98.9295%	higher
ED≤1 Type	88.0416%	96.9503%	much higher

(28-05-2026, 10:22 PM)Dunsel Wrote: You are not allowed to view links. Register or Login to view.And you see what I was saying about word length.  You have no 1, 2 or 3 length words so you distribution is way to the right.  Voynich is word length 5 centered, not 6.

Problem solved—not perfect, but good enough.

[attachment=15831]

============================================================
PAGE 49
============================================================

chedyol ckh otecheo otdy otain qok sheol choeeyky
ytchy ykhol okor cheolky oteydy okar chockhy ytol
oteydy ch oldain sheol daar sheol cheesey char
l olal ykol cholody okodar choolchdy qokshey shain
qokchy otalshy oka qoteshy ok daseky ira otair
oteydy aiirody qooraiin cheky chky oleeol ote shey
qotor aiindy oleey chody qotey kal ytal oldyol
ok qo otdy choty oldyol otockey dair otardar
sheekchey qokchedy sheey choindy qocphy qotey ykchy chteey
oteor rot ykeey qokychdy oteal che chchy qokeeydy
qo ch shcthy cheedalaiin aiolky qokaiiin otal aithar
ykchy okear dair oteor sheey qo aiiinol

============================================================
PAGE 50
============================================================

sheol oka ykeydy oka ykchy chear otey aiir
lol yld choolky aiir chear osh oka choldar
charain qochy cheor otdady ol ok shoky oleydy
qokeey ytal ol ytey shdy shkshy chty daiin
chty okoraiin otor chodaiin ollchy olardy darody yteey
ykar cheor ykeey daiin okedy okar okaiin qofaiin
shkear otosey daiin sheey oleydy okeey qokeey qoeeor
ol checphey qochey chindy ch shdchy cheokeor daiin
chey daiir ykhol oltaiin okol ch shedy shechy
chor che che chedy chtain shedy chekydy daiin
qot oteeo qoin qolchy okchy cheol qot oksheo
okshor shol ykdy qotarain okaiin qolshey cheol

(29-05-2026, 05:16 AM)bi3mw Wrote: You are not allowed to view links. Register or Login to view.

(28-05-2026, 10:22 PM)Dunsel Wrote: You are not allowed to view links. Register or Login to view.And you see what I was saying about word length.  You have no 1, 2 or 3 length words so you distribution is way to the right.  Voynich is word length 5 centered, not 6.

Problem solved—not perfect, but good enough.

Now you're looking much more Voynich and you have at least one metric in the ballpark.  I'm guessing adding those short words reduced all the long words and brought that chart back closer to reality.  You're now matching word length distribution for a natural language.  The Voynich tends to have slightly shorter words than a natural language.  And, that bell shaped standard distribution should also start to bring your zipf curve into line.

(29-05-2026, 10:52 AM)Dunsel Wrote: You are not allowed to view links. Register or Login to view.And, that bell shaped standard distribution should also start to bring your zipf curve into line.

[attachment=15835]

In conclusion, I would like to clarify once again that my model is based primarily (87.4%) on a table containing 485 entries (syllable combinations). The ledger words only fill in the gaps not covered by short_words and fixed words. They therefore account for only a small percentage, but they are necessary to ensure that all words are generated. Otherwise, the model as a whole would not function properly.

Proportions of word generation:
-----------------------------------------

Short words:  1634 (17.2%)
Common words:  4474 (47.1%)
Common combinations:  2195 (23.1%)
Ledger words:  1197 (12.6%)
Total: 9500

The high proportion of predefined words suggests that the author may indeed have worked with a tables book or combinatorics. While this would be labour-intensive, it would still be feasible for a 15th-century author.

Of course, that doesn't prove the manuscript is meaningless. A table book could also encode a real language or a cipher system.

[attachment=15847]

(26-05-2026, 04:11 PM)Dunsel Wrote: You are not allowed to view links. Register or Login to view.

(26-05-2026, 03:07 PM)Torsten Wrote: You are not allowed to view links. Register or Login to view.I prefer to analyze the text the scribe actually wrote rather than speculate about the texts he didn't write — since that number is endless.

I will politely disagree with this statement. A model does not just have to explain what the scribe wrote. It also has to explain why the text did not fall apart into garbage.  A lot of objections in these discussions are really variations of “why don’t we see this?” or “why didn’t the scribe do that?” They are questions about constraint.

In my own experiments, most copy/mutate systems fail pretty quickly unless the constraints are tight enough. They drift, bloat, repeat badly, or start producing obvious nonsense. And, given enough generations, even my generator has issues. That failure is informative. It tells us the Voynich text is not just “anything goes” mutation. So I think the unwritten text matters too. If a proposed method could easily generate endless bad Voynichese, then we need to explain why the manuscript consistently avoided doing exactly that.

If we don't explore the possibilities of what wasn't created then we'll never have acceptance for what is created.

You raise a fair point — but it's a different question from the one I answered. Zandbergen asked why some unspecified modifications are absent. I said: because the scribe took one path and not another, and we can only analyze the path taken. That stands.

Your question is different: why doesn't the text fall apart into garbage? That's a question about constraint, and it has a straightforward answer.

The copying process inherits structure from the source. Each new word is a modification of an existing word that already has valid structure. A modification of a valid word almost always produces another valid word — because the stroke-level constraints are preserved by the modifications taken. The structure doesn't need to be maintained by external rules — it's carried forward by the copying process itself.

And the scribe has eyes. If a modification produces something that looks wrong, he sees it and doesn't write it. Your own "don't look stupid guardrails" describe this exactly. The scribe isn't following a ledger. He's looking at what he produced and judging whether it looks acceptable. Human aesthetic judgment is the constraint. 

His visual judgment follows the stroke-level structure of the writing system. Schwerdtfeger described in 2008 four design rules: (1) line-glyphs can follow line-glyphs or 'a'; (2) curve-glyphs and 'a' can follow curve-glyphs; (3) the 'l'-glyph can be used as a curve-glyph or as a line-glyph; and (4) gallows glyphs count as curve glyphs (see Timm & Schinner 2020, p. 10). These aren't rules the scribe memorized — they're descriptions of what visual similarity produces. A scribe modifying strokes naturally stays within these constraints because crossing them would look wrong to his eye.

This is also why your generator struggles — and why any generator will. For the human mind it is difficult not to repeat itself. The variation comes from imperfect repetition — not from randomness. The consistency comes from habitual patterns — not from explicit rules. A computer needs both dice (for variation) and rules (for consistency) as separate mechanisms. A human produces both from one source — his own cognitive habits. That's why modeling a human scribe in Python is, as you put it, "a nightmare." You can't model the human mind like a computer — because the human produces variation and structure from the same process, while the computer needs separate mechanisms for each.

The manuscript doesn't fall apart because every word was seen and accepted by a human before it became part of the source pool for the next word. The constraints are the pattern recognition abilities and the habits of the scribe himself.

(30-05-2026, 07:30 PM)Torsten Wrote: You are not allowed to view links. Register or Login to view.The manuscript doesn't fall apart because every word was seen and accepted by a human before it became part of the source pool for the next word. The constraints are the pattern recognition abilities and the habits of the scribe himself.

I couldn't agree with you more. And again, that brings human psychology into the equation. Which patterns?  And then, how do you model those patterns?  At what point do the weights and rules become more than a 15th century scribe can do?  That's where I am now.

For examaple:

In the other post on <ed> I mentioned I did some machine learning, trying to find syllables.  I refined that down to where only 3 words in Takahashi and 14 in Zandbergen/Landini can't be mapped has having good, learnable n-grams in Scribe 1 content.  It came up with a count of 150 main n-grams and 50 more that that act as connectors.  I added that learned chunk data into my generator and it produced the code blocks I pasted below.  

The ledger model still attempts to do a mutation and validates it. The new word is then, it's passed off to the chunk model to see if it contains valid n-gram chunks with valid neighbors.  It looks more Voynich like that previous runs and still matches statistics pretty well. 

But, is this still human doable.  I think so but, it may require more than just the scribe's eyes . There may have been an audible component, where the scribes were familiar enough with the output tokens that they pronounced them.  I know I pronounce Voynich words to see if they sound right. I pronounce words as I write them.  Might a Voynich scribe do the same?


After generating 1000 pages, it starts to look less Voynich-like but still has familiar patterns.


And here's page 10,000.  I assumed if the model was going to collapse, it would have done so by this point.  I can't provide any stats to compare to Voynich other than to say it's a bit heavy on longer words which could be adjusted.

(30-05-2026, 04:28 PM)bi3mw Wrote: You are not allowed to view links. Register or Login to view.Of course, that doesn't prove the manuscript is meaningless. A table book could also encode a real language or a cipher system.

Ran some of my tests on your output.  Hope you don't mind a critical analysis.  

Test	His Output	Takahashi Slice	Comment
Tokens	9,447	9,447	Matched sample size
Types	1,757	2,739	Far fewer unique words
Hapax	683	1,856	Far fewer one-off words
Type/Token Ratio	18.6%	29.0%	More repetitive vocabulary
Hapax / Type Share	38.9%	67.8%	Major novelty deficit
Mean Word Length	5.04	4.87	Very close
Zipf Slope	-1.021	-0.841	Too steep. Common words dominate too much
Length Distribution JSD	0.0047	-	Excellent match
ED≤1 Token Hit	89.55%	-	Lower than expected
ED≤1 Type Hit	77.52%	-	Weak word-family structure
Bigram Reject Rate	2.94%	-	Reasonably good
Trigram Overlap	79.84%	-	Moderate match

My eyeballs say your text looks good, almost too good perhaps. The Voynich has it's own license on weird. Which I suspect is why your hapax and novelty are low.