- Options
- Login
- Desktop Style
Options
Aanother statistical take on Voynichese
- Aanother statistical take on Voynichese
-
RE: Aanother statistical take on Voynichese
nablator > 30-06-2026, 08:19 PM
In my experience, a 3rd-order Markov chain too frequently produces long words, such as cholchekaiiin, otocholsheedan, qokedalcheoly, sheolalsheeey, olkeechokodaiin.
EDIT:
Actually it was 2nd order (I just picked a random trigram from a transliteration starting with the 2 previously generated characters). I haven't tried 3rd order. -
RE: Aanother statistical take on Voynichese
Mauro > 30-06-2026, 10:18 PM
(30-06-2026, 06:17 PM)kostrubaty Wrote: You are not allowed to view links. Register or Login to view.As for indistinguishable -> It simply means that given 2 text excerpts one in voynichese one generated you're unable to tell which is which. You're able to tell only because you've checked with known text. If I were to say here's a new text I found in this language you could not objectively prove/disprove my claim basing your judgment solely on the text itself.
If you can link me a reasonably long text sample generated by your Markov chain, say at least ten thousand words, I can run a battery of tests and tell how it compares to the original Voynich and to previous pseudo-Voynichese texts. -
RE: Aanother statistical take on Voynichese
JoeyB > 01-07-2026, 12:53 AM
(30-06-2026, 08:09 PM)oshfdk Wrote: You are not allowed to view links. Register or Login to view.
(30-06-2026, 08:03 PM)Jorge_Stolfi Wrote: You are not allowed to view links. Register or Login to view.For a Markov of order 3 the Author would need a table with ~20x20 = 400 entries listing the alternatives for the next character given the previous 2, each with its probability.
I thought order 3 is based on three previous entries? 20x20x20?
Correct, the code line in his repo You are not allowed to view links. Register or Login to view. (the --order default is at You are not allowed to view links. Register or Login to view., the ^^^ padding at You are not allowed to view links. Register or Login to view.)):
41: for (let i = order; i < s.length; i++) {
42: const ctx = s.slice(i - order, i) // context = the `order` chars before position i
43: const ch = s[i]!
With order = 3, s.slice(i - 3, i) is the 3 characters immediately before the one being predicted. So the context is the previous 3 characters, which is 20×20×20…
Then these lines in the same file:
generate.ts:18 sets the default: const order = ... ?? 3.
generate.ts:37 const PAD = "^".repeat(order) pads each word with three ^, so the first prediction already has a 3‑char context.
The generator then has: generate.ts:60 let ctx = (PAD + seed).slice(-order) and generate.ts:65 ctx = (ctx + ch).slice(-order) both keep the last 3 characters as state. -
RE: Aanother statistical take on Voynichese
nablator > 01-07-2026, 10:29 AM
(30-06-2026, 10:18 PM)Mauro Wrote: You are not allowed to view links. Register or Login to view.
(30-06-2026, 06:17 PM)kostrubaty Wrote: You are not allowed to view links. Register or Login to view.As for indistinguishable -> It simply means that given 2 text excerpts one in voynichese one generated you're unable to tell which is which. You're able to tell only because you've checked with known text. If I were to say here's a new text I found in this language you could not objectively prove/disprove my claim basing your judgment solely on the text itself.
If you can link me a reasonably long text sample generated by your Markov chain, say at least ten thousand words, I can run a battery of tests and tell how it compares to the original Voynich and to previous pseudo-Voynichese texts.
I tried the generator.
I have node.js already installed (with npm), but this TypeScript code needs bun.
To install it:
npm install -g bun
There is an option --show to output any number of generated words.
So for example:
bun generate.ts --section=recipes --show=11642
displays:
Code:real corpus: 11642 words | Markov order 3 | recipes generated: 11642 words metric REAL GENERATED ---------------------------------------------------- char H2 (bits) 2.02 2.02 char H3 (bits) 1.75 1.78 word len mean 5.19 5.20 word len var/mean 0.63 0.75 type/token ratio 0.281 0.271 hapax % of types 68.8 67.3 immediate-repeat % 0.72 0.47 self-citation 0.676 0.648 ---------------------------------------------------- suffix(2) profile cosine real↔gen: 1.000 (1.0 = same inventory) prefix(2) profile cosine real↔gen: 1.000
and outputs 11642 generated words:
gen_recipes.txt (Size: 70.29 KB / Downloads: 8)
-
RE: Aanother statistical take on Voynichese
ololololo > 01-07-2026, 11:23 AM
(01-07-2026, 10:29 AM)nablator Wrote: You are not allowed to view links. Register or Login to view.
The result is impressive. Although many words in the manuscript do not appear, they seem to be "correct" from her perspective (such as olchaim).(30-06-2026, 10:18 PM)Mauro Wrote: You are not allowed to view links. Register or Login to view.
(30-06-2026, 06:17 PM)kostrubaty Wrote: You are not allowed to view links. Register or Login to view.As for indistinguishable -> It simply means that given 2 text excerpts one in voynichese one generated you're unable to tell which is which. You're able to tell only because you've checked with known text. If I were to say here's a new text I found in this language you could not objectively prove/disprove my claim basing your judgment solely on the text itself.
If you can link me a reasonably long text sample generated by your Markov chain, say at least ten thousand words, I can run a battery of tests and tell how it compares to the original Voynich and to previous pseudo-Voynichese texts.
I tried the generator.
I have node.js already installed (with npm), but this TypeScript code needs bun.
To install it:
npm install -g bun
There is an option --show to output any number of generated words.
So for example:
bun generate.ts --section=recipes --show=11642
displays:
Code:real corpus: 11642 words | Markov order 3 | recipes generated: 11642 words metric REAL GENERATED ---------------------------------------------------- char H2 (bits) 2.02 2.02 char H3 (bits) 1.75 1.78 word len mean 5.19 5.20 word len var/mean 0.63 0.75 type/token ratio 0.281 0.271 hapax % of types 68.8 67.3 immediate-repeat % 0.72 0.47 self-citation 0.676 0.648 ---------------------------------------------------- suffix(2) profile cosine real↔gen: 1.000 (1.0 = same inventory) prefix(2) profile cosine real↔gen: 1.000
and outputs 11642 generated words: -
RE: Aanother statistical take on Voynichese
oshfdk > 01-07-2026, 11:25 AM
(01-07-2026, 10:29 AM)nablator Wrote: You are not allowed to view links. Register or Login to view.and outputs 11642 generated words:
To my eye about one word in 6 looks weird. Just to make it fair, I didn't cross check with the transliterations, I've just marked below the words that I see as strange. I think there is nothing really wrong with most these words and I bet many of them do appear in the MS occasionally, but generally they look like rare oddities and this text is packed with them. Also, I'm not sure the end/start pairs (across spaces) look like those in the manuscript.
sor chedy chol kas chdor saiin oky alody kchy qokey qokalkaraiin ychshodar qokeey lkeey shy chteeey qoeear y olkeey chdar oleedy chee okchedy chockhey sain cheeol chey dcheedy qoeear okain yk daiin lteor sotchody aiin chsd otam qotarod dy chdal kaiin okain olkaiin lchedy chokoeey oshedy chol shey lr qotaly lkeeey cholaiin cheedy oiin otal okain lkeeey sheodain pchey lor qokeedy odaiin qo ls shal al okcheealkam otain qotain lotal qokchodain otain chees olkeeedy al otedy okair al cheo ain teey alchedy okaiidy qoiir lol chckhy chedy checkhy sheek okcheds oky qol shoror cheky dsheor qokshy qoeeey chedy qokam lag she f oteey okchdy chey okal qol chain opaiin otyd shoky qoekeo daiin okeedy aiin lody al aiin shedy alaiin pchedy aiy qekor oteey aiin okeedodar -
RE: Aanother statistical take on Voynichese
nablator > 01-07-2026, 12:08 PM
(01-07-2026, 11:25 AM)oshfdk Wrote: You are not allowed to view links. Register or Login to view.To my eye about one word in 6 looks weird. Just to make it fair, I didn't cross check with the transliterations, I've just marked below the words that I see as strange. I think there is nothing really wrong with most these words and I bet many of them do appear in the MS occasionally, but generally they look like rare oddities and this text is packed with them.
The generator uses only the "Recipes" section of ZL3b-n.txt for training the model. It should not produce any character 4-gram that doesn't exist in the "Recipes" (Q20) section.
There are 2 ".kas" in ZL3b-n.txt, but they are not in Q20. Weird.
EDIT:
There is ".dal,kasol." in f103r.13 and ".okas." in f115v.40.
Since "." and "," are equivalent:
s = s.replace(/,/g, ".") // uncertain space → word break
".kas." is made of 2 existing 4-grams.
Quote:Also, I'm not sure the end/start pairs (across spaces) look like those in the manuscript.
The words are generated separately in a loop calling genWord(), not taking into account the previously generated words (except when copying or mutating a previous word). With seed = "", ctx is always the same (= PAD).
Code:const PAD = "^".repeat(order) ... function genWord(seed = ""): string { let w = seed let ctx = (PAD + seed).slice(-order) -
RE: Aanother statistical take on Voynichese
Mauro > 01-07-2026, 01:01 PM
I compared some basic statistics of four texts:
- The original VMS (transcription RF1a-n with dubious characters removed)
- A pseudo-Voynich text made with a program I wrote some time ago (Asemic_LOOP), generated with an order-1 Markov chain, 1201 states, 8115 transitions
- The file posted by nablator in You are not allowed to view links. Register or Login to view., kostrubaty's generator, order-3 Markov chain, ?? states, ?? transitions
- The output of Timm Torsten's generator (on github), copy-and-modify mechanism
These are the results (tables ordered by distance from the original VMS):
Monograms/bigrams statistics. Min possible value = 0, max possible value = sqrt(2) =~ 1.414
Statistics of following/preceding character. Min possible value = 0, max possible value = long to explain, above 3
Statistics of words distribution (vocabulary, Zipf's law and words length). Min possible value = 0, max possible value = sqrt(2)
kostrubaty's generator does a reasonably good job in emulating Voynichese, sitting +- midway between my Asemic_LOOP and Timm Torsten's copy&modify. If you want I can also post about entropies and hapax legomena, but it'll take me some time to format the output. -
RE: Aanother statistical take on Voynichese
kostrubaty > 01-07-2026, 05:27 PM
Okay, sorry I don't have time right now to respond to everyone properly, so I'll try to address all your notes.
Firstly there's an updated version, I've included some larger samples you can use if you don't want to install bun.
Also you're right that markov chain in itself would be unwieldy, so I worked in that direction a bit which produced this proof of concept
You are not allowed to view links. Register or Login to view.
You'll notice it's much simpler, you still need gear mechanism, or some markings that line up indicating valid combination.
And according to the entropy distribution it doesn't need to be complicated at all.
This could also be presented in some form of decision graph, but I've not done any research into how this could look like in middle ages.
Either way, they could easily create multiple simply by copying. If they introduced some mistakes while prepping copies this opens a slightly different front on the source analysis.
However this is entirely possible that the bias is purely introduced by human.
Either way it'd need much more detailed work to verify the results. I might look more into this at some later time, however feel free to use all of that in any way you want. If it leads to something, a mention would be nice.
Also note, all of the code and the text in the repo is generated by claude under my direction. I'm not a native english speaker nor a real researcher - so this was the only way for me to quickly set this up. However methodology should be solid. Code is simple and mostly self explainable, for me it was easier to do with typescript instead of python. I've removed bun dependency so you can now run it with just node 24.
If you have any specific questions let me know.
You are not allowed to view links. Register or Login to view.
You are not allowed to view links. Register or Login to view.