• Options
• Login
• Desktop Style

Aanother statistical take on Voynichese

• Index

  • The Voynich Ninja
  • Voynich Research
  • Theories & Solutions
  • Aanother statistical take on Voynichese
• Aanother statistical take on Voynichese

• Aanother statistical take on Voynichese

  kostrubaty > Yesterday, 05:13 AM

  So here's my analysis
  
  You are not allowed to view links. Register or Login to view.
  
  Results strongly indicate that the text was generated via simple methodology (order-3 markov chain).
  Also this shows that there is zero measurable correlation between text and the imagery. 
  Also text written by different "hands" all matches statistical fingerprint, which strongly suggests use of shared methodology for text generation.
  
  Does it prove anything? Guess not, however result seem to leave little to imagination.
  
  All the code necessary for replicating results, detailed findings and vl dataset is in the repo.

• RE: Aanother statistical take on Voynichese

  oshfdk > Yesterday, 12:12 PM

  I'm not sure order-3 Markov chain is "simple". Can you explain how would the XV century scribes implement it? For starters, this requires some well designed state transitions, and a lot (thousands?) of them. It's one thing to approximate some sequence of characters using a Markov chain, it's a different thing to explain how this sequence of characters was produced.
  
  Also, your generated sample, which you describe as "indistinguishable from Voynichese to the eye" appears a bit odd to me:
  
  qoeeal shd shos lsheodain oteedy dchdar ol ar okold qokal qopchdy or olkchy qokeey yteey otair deeol shar chedy chor ykodaiin
  
  It's reasonably good, which is of little surprise given the order-3 Markov chain, however I don't think qoeeal looks very plausible (it doesn't appear in the MS), and neither does lsheodain or ykodaiin, though they look less iffy, while okold doesn't appear as a standalone word, dchdar is quite rare, deeol only appears once as a standalone word and once as a prefix. So, to my eye this looks like very unlikely Voynichese, because it's packed with rare and unusual words.

• RE: Aanother statistical take on Voynichese

  Jorge_Stolfi > Yesterday, 01:27 PM

  (Yesterday, 12:12 PM)oshfdk Wrote: You are not allowed to view links. Register or Login to view.

  (Yesterday, 05:13 AM)kostrubaty Wrote: You are not allowed to view links. Register or Login to view.So here's my analysisn You are not allowed to view links. Register or Login to view.
  Results strongly indicate that the text was generated via simple methodology (order-3 markov chain).
  Also this shows that there is zero measurable correlation between text and the imagery.  Does it prove anything? Guess not, however result seem to leave little to imagination.

  your generated sample, which you describe as "indistinguishable from Voynichese to the eye" appears a bit odd to me:
  qoeeal shd shos lsheodain oteedy dchdar ol ar okold qokal qopchdy or olkchy qokeey yteey otair deeol shar chedy chor ykodaiin
  It's reasonably good, which is of little surprise given the order-3 Markov chain

  
  Indeed.  An order-3 Markov can probably generate also text that will be "indistinguishable from English to the eye" (of someone who cannot read English).  Here is the output of Claude Shannon's classic experiment: "IN NO IST LAT WHEY CRATICT FROURE BIRS GROCID. PONDENOME OF DEMONSTURES OF THE REPTAGIN IS. REGOACTIONA OF CRE"
  
  An order-3 Markov will reproduce the frequency distributions of characters, bigrams, and trigrams of the training set.  Thus it will reproduce the character entropies H0, H1, and H2, and any correlations of characters across word gaps.  It will also approximate correlations over longer distances, because of chained correlations (the 4th letter depends on letters 2 amd 3, but these depend on letter 1, so letter 4 will partly depend on letter 1 too.)  IIRC, there is a good chance that it will produce output with a Zipf-like word frequency distribution.
  
  An order-3 Markov may fail to capture some subtler properties of the training set, such as the distribution of word type lengths or a rule like "a word can have at most one letter 't'  or 'k', but not both".  But the "core-mantle-crust" structure of Voynichese words makes it possible for an order-3 Markov to "learn" the rule "a word can have at most one gallows". That's because the last 3 glyphs will almost always tell the generator whether it is in the "prefix" part ("uphill", before the gallows) or in the "suffix"  part ("downhill", after the gallows); and thus it can avoid generating a second gallows in the latter.  
  
  On the other hand, an order-3 Markov may miss the rules "at most three benches" (Ch, Sh, ee; counting platform gallows like CTh) and "at most three dealers" (d, l, r, s) that seem to hold for Voynichese words.  Do we see exceptions to these rules in the output of your generator?
  
  All the best, --stolfi

• RE: Aanother statistical take on Voynichese

  JoeyB > Yesterday, 02:06 PM

  Nice, thanks for sharing all of that esp. the githib repo. FWIW, when I have played with Claude in past I watched it talk itself out of the markov-4 story when I pressed it with two things: first, it basically broke wqhen I forced it to adversarially explain Tavie's vertical impact effect. That thread is active again now but its basically that the first glyph of a line is conditioned on the first glyph of the line above it and it avoids repeating that glyph (and then there's directed transitions right, o->q, y->o, etc). But the left to right character chain Claude is proposing has no page in it, the models I played with didnt have a variable for "look at the line above" and it was only checking reading order. Tavie's work and the threads on this were gold for beating up the linear model.
  
  Second thing I had trouble with was figuring out if the model was really finding coupling or was just memorizing. Claude got very excited that it had reproduced coupling early but then it would turn out it was just handing the samples back, IIRC the threads on generators and held-out vocabulary were super helpful in pressing this point.
  
  Good luck!

• RE: Aanother statistical take on Voynichese

  kostrubaty > 9 hours ago

  (Yesterday, 12:12 PM)oshfdk Wrote: You are not allowed to view links. Register or Login to view.I'm not sure order-3 Markov chain is "simple". Can you explain how would the XV century scribes implement it? For starters, this requires some well designed state transitions, and a lot (thousands?) of them. It's one thing to approximate some sequence of characters using a Markov chain, it's a different thing to explain how this sequence of characters was produced.
  
  Also, your generated sample, which you describe as "indistinguishable from Voynichese to the eye" appears a bit odd to me:
  
  qoeeal shd shos lsheodain oteedy dchdar ol ar okold qokal qopchdy or olkchy qokeey yteey otair deeol shar chedy chor ykodaiin
  
  It's reasonably good, which is of little surprise given the order-3 Markov chain, however I don't think qoeeal looks very plausible (it doesn't appear in the MS), and neither does lsheodain or ykodaiin, though they look less iffy, while okold doesn't appear as a standalone word, dchdar is quite rare, deeol only appears once as a standalone word and once as a prefix. So, to my eye this looks like very unlikely Voynichese, because it's packed with rare and unusual words.

  Well if you look at the values the entropy is ~2 that means that characters are chosen from very limited pool, this means that given a character you only have 2 options to choose for the next one. They could have used some form of volvelle with cuotouts, but it can be just a simple flow diagram.
  As for indistinguishable -> It simply means that given 2 text excerpts one in voynichese one generated you're unable to tell which is which. You're able to tell only because you've checked with known text. If I were to say here's a new text I found in this language you could not objectively prove/disprove my claim basing your judgment solely on the text itself.
  
  This actually neatly explains why is the structure like this, they probably prepared the cartouches by prefilling first letter of each line. 
  Then scribe would use the diagram to look up the text and write it down letter by letter. 
  
  Other characteristics like self repetition are simply caused by human error. Imagine sitting in a dim workshop writing down sequences of nonsensical letters. At some point you'll just start repeating the words sometimes just to avoid looking them up on the diagram. 
  
  This also explains how 5 different authors could produce text with exactly same statistical fingerprint which is unrealistic even if we assume it's a translation of a source material.

• RE: Aanother statistical take on Voynichese

  kostrubaty > 9 hours ago

  (Yesterday, 01:27 PM)Jorge_Stolfi Wrote: You are not allowed to view links. Register or Login to view.

  (Yesterday, 12:12 PM)oshfdk Wrote: You are not allowed to view links. Register or Login to view.

  (Yesterday, 05:13 AM)kostrubaty Wrote: You are not allowed to view links. Register or Login to view.So here's my analysisn You are not allowed to view links. Register or Login to view.
  Results strongly indicate that the text was generated via simple methodology (order-3 markov chain).
  Also this shows that there is zero measurable correlation between text and the imagery.  Does it prove anything? Guess not, however result seem to leave little to imagination.

  your generated sample, which you describe as "indistinguishable from Voynichese to the eye" appears a bit odd to me:
  qoeeal shd shos lsheodain oteedy dchdar ol ar okold qokal qopchdy or olkchy qokeey yteey otair deeol shar chedy chor ykodaiin
  It's reasonably good, which is of little surprise given the order-3 Markov chain

  
  Indeed.  An order-3 Markov can probably generate also text that will be "indistinguishable from English to the eye" (of someone who cannot read English).  Here is the output of Claude Shannon's classic experiment: "IN NO IST LAT WHEY CRATICT FROURE BIRS GROCID. PONDENOME OF DEMONSTURES OF THE REPTAGIN IS. REGOACTIONA OF CRE"
  
  An order-3 Markov will reproduce the frequency distributions of characters, bigrams, and trigrams of the training set.  Thus it will reproduce the character entropies H0, H1, and H2, and any correlations of characters across word gaps.  It will also approximate correlations over longer distances, because of chained correlations (the 4th letter depends on letters 2 amd 3, but these depend on letter 1, so letter 4 will partly depend on letter 1 too.)  IIRC, there is a good chance that it will produce output with a Zipf-like word frequency distribution.
  
  An order-3 Markov may fail to capture some subtler properties of the training set, such as the distribution of word type lengths or a rule like "a word can have at most one letter 't'  or 'k', but not both".  But the "core-mantle-crust" structure of Voynichese words makes it possible for an order-3 Markov to "learn" the rule "a word can have at most one gallows". That's because the last 3 glyphs will almost always tell the generator whether it is in the "prefix" part ("uphill", before the gallows) or in the "suffix"  part ("downhill", after the gallows); and thus it can avoid generating a second gallows in the latter.  
  
  On the other hand, an order-3 Markov may miss the rules "at most three benches" (Ch, Sh, ee; counting platform gallows like CTh) and "at most three dealers" (d, l, r, s) that seem to hold for Voynichese words.  Do we see exceptions to these rules in the output of your generator?
  
  All the best, --stolfi

  
  Yes it does produce Zipf like distribution. Crucial information is that voynichese it is sequential not structural.  So this led me straight to simple order-3 markov, however with pure markov the distributions were slightly different. After adding self repetition layer to the generation method at ~0.15 chance the stats just clicked in. 
  
  As for the gallows - this model simply ignores any specific rules that might or might be there. The goal was to check if a simple, reproducible method existst that would give the text with this exact characteristics. 
  
  While it does not prove anything it simply gives a perfectly valid and reasonable explanation of how this was achieved.

• RE: Aanother statistical take on Voynichese

  kostrubaty > 9 hours ago

  (Yesterday, 02:06 PM)JoeyB Wrote: You are not allowed to view links. Register or Login to view.Nice, thanks for sharing all of that esp. the githib repo. FWIW, when I have played with Claude in past I watched it talk itself out of the markov-4 story when I pressed it with two things: first, it basically broke wqhen I forced it to adversarially explain Tavie's vertical impact effect. That thread is active again now but its basically that the first glyph of a line is conditioned on the first glyph of the line above it and it avoids repeating that glyph (and then there's directed transitions right, o->q, y->o, etc). But the left to right character chain Claude is proposing has no page in it, the models I played with didnt have a variable for "look at the line above" and it was only checking reading order. Tavie's work and the threads on this were gold for beating up the linear model.
  
  Second thing I had trouble with was figuring out if the model was really finding coupling or was just memorizing. Claude got very excited that it had reproduced coupling early but then it would turn out it was just handing the samples back, IIRC the threads on generators and held-out vocabulary were super helpful in pressing this point.
  
  Good luck!

  
  No probs, I've used claude too for the statistical code, however I was actually trying to find just about any correlation between text and the images. Unfortunately  there was nothing, as in completely absolutely nothing. So based on that either the text is just jibberish or perhaps there's something we don't understand.
  
  Bit disappointing, however a story about middle ages merchant producing a fake book that keeps eluding people 500 years later is quite a story too. 
  
  It's just about finding a simplest plausible method at this point. So unless someone can prove it could NOT have been created that way I'll personally treat that as demystified.

• RE: Aanother statistical take on Voynichese

  oshfdk > 8 hours ago

  (9 hours ago)kostrubaty Wrote: You are not allowed to view links. Register or Login to view.As for indistinguishable -> It simply means that given 2 text excerpts one in voynichese one generated you're unable to tell which is which. You're able to tell only because you've checked with known text.

  
  I can't agree. I think it's likely the text that this system generates is clearly and unmistakably different from Voynichese, and the larger the sample the more obvious this will become. Unfortunately, you have only provided a very short sample, but it doesn't look very plausible to me.
  
  

  (9 hours ago)kostrubaty Wrote: You are not allowed to view links. Register or Login to view.This actually neatly explains why is the structure like this, they probably prepared the cartouches by prefilling first letter of each line. 
  Then scribe would use the diagram to look up the text and write it down letter by letter. 

  
  What is the size of the Markov model? Suppose you represent it as a table, and you need a mapping from a trigram to possible next characters, which means the table is at least at long as the number of different trigrams in the MS. The number of different repeating trigrams in the text is approximately 2500, if I'm not mistaken. This is some 10+ full text pages. Basically, a separate manuscript is needed to implement this scheme.

• RE: Aanother statistical take on Voynichese

  Jorge_Stolfi > 8 hours ago

  (9 hours ago)kostrubaty Wrote: You are not allowed to view links. Register or Login to view.Well if you look at the values the entropy is ~2 that means that characters are chosen from very limited pool, this means that given a character you only have 2 options to choose for the next one. They could have used some form of volvelle with cuotouts, but it can be just a simple flow diagram.

  
  For a Markov of order 3 the Author would need a table with ~20x20 = 400 entries listing the alternatives for the next character given the previous 2, each with its probability.  
  
  

  Quote:As for indistinguishable -> It simply means that given 2 text excerpts one in voynichese one generated you're unable to tell which is which.

  
  You mean: "a person who knows nothing about the Voynichese script would be unable to tell which is which".
  
  By the same method and arguments, one could "show" that Hamlet could have been generated by a 3-Markov process.  Just show the two texts, original and synthetic, to a Binumarien speaker who knows nothing of English (not even the alphabet) and see if he can tell which is which. 
  
  All the best, --stolfi

• RE: Aanother statistical take on Voynichese

  oshfdk > 8 hours ago

  (8 hours ago)Jorge_Stolfi Wrote: You are not allowed to view links. Register or Login to view.For a Markov of order 3 the Author would need a table with ~20x20 = 400 entries listing the alternatives for the next character given the previous 2, each with its probability.  

  
  I thought order 3 is based on three previous entries? 20x20x20?
• Next Oldest Next Newest