Voynich words evolve over a linear sequence

[JoJo_Jost]

Following up on my earlier post about the tables: I have now tested this against your sequence.

I entered your sequence (entries from the SVG export of the Sequence tab, 207 pages) into my tables and examined a single aspect: the ending after the root, i.e., chy / chey / chedy / cheedy, etc. The graph shows all words ending in -y, split by the ending (-y, -ey, -edy …) and the glyph before it (ch, sh, k, t …). In my own notation this is the table “State x Root” for key y.

The result was even more unambiguous than I had expected. The forms with a “d” after the “e” run (chedy, shedy, kedy, tedy, keedy, teedy) are practically nonexistent up to rank 124, i.e. the first 124 pages of your sequence (Herbal A, Pharma f87–f102, Astro f67–f70): 65 instances out of 4,215 “y” tokens. Starting at rank 125 (f46r), they appear immediately and together account for over a third of all y-tokens (3,978 out of 10,642). In between, I can’t detect any gradual increase. Pages like f99r, f101r, or f102v2 each have 30 to 40 “e-runs” and not a single “d”; the first pages after the jump immediately reach 60 to 85%.

I find two things about this important and interesting.

First, the new rows in the table duplicate existing ones. “chedy” is distributed across the preceding glyphs almost exactly like “chey” (ch/sh-heavy), and “cheedy” almost exactly like “cheey” (k-heavy). No new pattern emerges; rather, it’s a second spelling for an existing one. The bulk of this comes primarily from “chy,” which drops from 14% to 3%. No row or column disappears.

In short: To me, this looks like a cipher table that has been expanded, not like drift.

Second, and this is particularly intriguing: There is (at least) one page where the change occurs right in the middle of the page: f57r. The first paragraph (lines 1–5) has 13 forms ending in -eody and none ending in -edy, while the second paragraph (starting on line 6, which begins with a gallows) has none ending in -eody and 10 ending in -edy.
So was the table altered and then presented to the scribes in the middle of their work??? We don’t know, we don’t know.
This also fits your EODY_EDY rule exactly, if I understand it correctly—but not distributed across sections, rather at a paragraph boundary. “Lost leaves” can’t really explain this at this point, can they?

But I should be careful—your sequence is partly built from transitions that include “ed”, so the position of the jump is not independent. What is independent: the pages before it have really zero, not just few, and You are not allowed to view links. Register or Login to view. shows the switch within one page. So far, it’s just one key; I haven’t looked at the others yet. The 20 zodiac and ring pages without positions aren’t included either.

   

[pjburkshire]

Who or what is "Golf"?  The Voynich Search Engine does not seem to know either...

Thanks to You are not allowed to view links. Register or Login to view. .

You are not allowed to view links. Register or Login to view.

Note, this classification scheme is referred to as 'Golf Regimes' in the app, this is my internal name, resulting from my habit of naming workstreams Alpha, Bravo, Charlie, etc.

[Jorge_Stolfi]

Who or what is "Golf"?  The Voynich Search Engine does not seem to know either...

Thanks to You are not allowed to view links. Register or Login to view. .

You are not allowed to view links. Register or Login to view.

Thanks!  

But, funny, I jasked voynicharchive.com earlier this morning and it said that there was no mention of "Golf" in Voynich.ninja" -- and no user @npcompl33t...  

All the best, --stolfi

[npcompl33t]

I am still trying to parse all the data you have produced.

Thank you for taking a look, I appreciate your feedback.

My first impression is that there may be a bit of circularity in your approach.

This is fair, I think I addressed this in my response to you in the chronology thread, where i showed that even using Currier x Section shows the same linear drift between steps.

Going back to the book sorting example, if you sort a hundred random books by height, you will find that the width also seems to gradually vary as you move along that sequence

This goes back to the horseshoe pattern discussion a bit, but in your example you are just sorting on one or two dimensions. I looked across all dimensions. If many dimensions move together, there is a real underlying factor. In the book example, if a publisher changed ther height and width of their books gradually each year, you would indeed find this ordering using the same methods I've been describing.

The fact that some of the word replacement rules are reversed at some point suggests that the ordering in fact is not meaningful. 

It doesn't suggest the ordering is not meaningful. Reversals are exactly what you would expect with changes that are random drifts.

H0. "The variations in the text statistics from section to section are essentially all determined by changes of topic, text format, text structure, and style."

Herbal A and B are the same topic, format, and style, and yet are significantly different. This was the same difference you and I had discussed and was the motivation behind this analysis.

H1. "The pages within each section are statistically homogeneous, and can be well modeled as random Gaussian-like deviations from a single central word frequency distribution."

The pages within each section are not stastically homogenous, even if we fully exclude Herbal A and B from this. I found meaningful splits in stars and Herbal A. Pages within a single bifolia can be modeled as H1 describes, and when doing so virtually all pages within i bifolio occur within 1.5% of each other on my identified time axis.

The forms with a “d” after the “e” run (chedy, shedy, kedy, tedy, keedy, teedy) are practically nonexistent up to rank 124,

This is an accurate observation, but I want to point out that we are missing multiple bifolia that probably would have occured prior to this section, and there is a gap in the overall chronolgy here. f58 and some astro prose shows indications of a transition, in particular, dy usage actually starts there, which would appear to be a precursor of the edy change,  so I still think that we can't say for certain if the change is as abrupt as it appears, or if it is a result of the missing sections.

First, the new rows in the table duplicate existing ones. “chedy” is distributed across the preceding glyphs almost exactly like “chey” (ch/sh-heavy), and “cheedy” almost exactly like “cheey” (k-heavy). No new pattern emerges; rather, it’s a second spelling for an existing one. The bulk of this comes primarily from “chy,” which drops from 14% to 3%. No row or column disappears.

I suspect that the exact transformation path one assumes here will dramatically change some reesults. I think chey and chy are probably related, they exhibit an almost flat frequency profile when examiend together. I suspect chol->cheol-> chedy and chor->cheor-> chedy. This is similar but slightly different than Torsten's chain which groups chol, cheol, cheo, chey all together as they are within a few edit distance of each other. The differences here are dramatic, as i will explain in my follow-up post below. Grouping words like Torsten's proposal, based on edits, generally preserves frequency distributions at the cost of destroying information. Different routes can be taken that sacrifice frequency preservation for information retention.

[pjburkshire]

Thanks!  

But, funny, I jasked voynicharchive.com earlier this morning and it said that there was no mention of "Golf" in Voynich.ninja" -- and no user @npcompl33t...  

What can I say?  Computers like me.  They are always trying to make me happy.

[npcompl33t]

The Emerging Question

The question here ultimately is going to be " is it possible to connect words from 1a all the way through to Bio in a way that preserves information and frequency". Or more plainly is an evolution sequence incompatible with preserving semantic meaning.

This is ultimately the debate between semantic meaning and self-citation. The evolution sequence or gradient alone does not mean there can be no semantic meaning. The question is whether the transformations destroy information or whether they preserve it.

I'm testing how much information survives between words at the start of the network, all the way through to the end, using the same golf sequence clusters (1a, 1b, 1.5a etc) as nodes along the path. It needs to tracks both frequency mass and information.


The goal is to see:

• How much of the vocabulary can survive the overall transformation chain, and how much pure information is conserved
  
• what 'preserved' information looks like concretely, in terms of merged forms
  
• how important the specific connection model is to this preservation, whether different models result in drastically different rates 
  

Simple First Method

I made a quick simple model to simulate this process, that tracks information merges across each step like

• if chol -> chedy and chor-> chedy then chedy = (50% chol, 50%chor)
  
• if chol + shol + qol = ol , ol = (33% chol, 33% shol, 33% qol)
  

This would be something comparable to:

• he + she + they = it , it = (33%he, 33%she, 33%they)
  

So in the pronoun example, information is lost, but it keeps track of how many words get blurred together. If he / she still appears a few times outside of the merge, it may be that information is still preserved in contextually relevant situations.

I am tracking both overall frequency mismatches, as well as information preservation, across several different linkage methods between sections.

Initial Tests

In my initial tests, allowing merges based on edit distances destroys information, exactly as you would expect under Torsten's theory, while also allowing for almost perfect frequency preservation. This isn't unexpected, it means that its easy to find a path through the chain using generic edit distance links that preserves frequencies, at the cost of destroying most of the information. 

• 1 edit distance transformations results in 65% information retention, with a final usage mismatch of only 17% (0% = perfect frequency match)
  
• 2 edit distance has only a 3% usage mismatch but information preservation drops to 49% (all words) and (45% common words)
  
• 3 edit distance has a 1.5% usag mismatch, only 44% of all word information and 37% of common word information is retained
  

My first simple path optimizer was able to find some paths that preserve 94% of word information, but can't seem to achieve better than 50% final usage mismatch.

This results for example with chedy containing the following ( chol 49.9%, chor 44.0%, cheol 4.1%, cheor 2.0% · <0.1% other values).

Vulgate Comparisons

However, when i test Vulgate using similar metrics, i get an average usage mismatch of 58.8%, a common word frequency mismatch of 24.5%, while still preserving 95% of information. Basically, vulgate combines words with different frequency profiles into a single token, causing a high frequency mismatch, while ultimately still preserving 95% of total information. This puts it roughly in the range of my initial simple voynich tests, close enough to justify more testing.


The New Bar

The bar is set. 

• A simplified version of Torsten's model allowing edit-distance based links preserves frequency mass almost perfectly, while destroying over half of information in the process. Note, this does not actually completely rule out semantic meaning. In this case, it would be extremely degenerative, but about 37-50% of information is preserved, depending on how relaxed the rules are. This still could be some type of degenerative or poor abbreviation system, or perhaps later LFD hands did not actually understand Voynich and did not effectively reproduce LFD hand 1's system
  
• It is possible to preserve information at the cost of frequency matching. Initial tests suggest this can still roughly fit within the expected values of an abbreviation system like vulgate.
  

The goal is now to try to determine if a 'path' can beat the null / edit distance model. Any solutions or mappings will need to prove they can work across the entire evolution sequence. 

We should try to see if we can locate the path that appears to make the most sense, evaluate if it preserves meaning and keeps frequency mismatch within acceptable ranges. I'm working on building out a more complex path optimizer, You are not allowed to view links. Register or Login to view..

Ideally we need to also do some more work on historical vulgate controls to help refine the range of values that may be acceptable.

[BrianCham1994]

Good work. Before I last left the field, I wanted to investigate this same pattern, to verify the evolution and "replacement rules" for words and glyph combos across the manuscript.

My working hypothesis was that text was generated/encrypted using paper volvelles (note the plural). These would inevitably wear out and literally fall apart from repeated rotations, so these would be replaced by new volvelles. These new volvelles would still satisfy the original system, but would not necessarily have the exact same glyphs/combos. Thus, there would be abrupt points where certain words and glyph combos would disappear, appear or change frequency.

For example (just a toy example, not realistic):

Volvelle 1 has:

• Ring 1 (outer): qo, __, o, __, qo, __, o, etc.
  
• Ring 2 (next one in): t, __, d, __, t, __, d, __, t, etc.
  
• Ring 3: ai, __, ai__, ai, __, ai, __, ai, etc.
  
• Ring 4: in, n, in, n, in, etc.
  

Where ___ means a blank space.
This can give you words like daiin, dain, aiin, qodaiin, qotaiin, etc.

Volvelle 2 has:

• Ring 1: qo, __, o, __, qo, __, o, etc.
  
• Ring 2: ch, __, sh, __, ch, __, sh, __, etc.
  
• Ring 3: ee, __, e, __, ee, __, e, etc.
  
• Ring 4: dy, y, dy, y, dy, y, etc.
  

This gives you words like chedy, shedy, qochedy, oshedy, y, etc.

Text is generated using volvelles 1 and 2 simultaneously to make words (realistically there'd be about five volvelles at any given time).
Then volvelle 1 wears out and is replaced by volvelle 1b, which is similar but not the same:

• Ring 1 (outer): o, __, o, __, o, __, o, etc.
  
• Ring 2 (next one in): k, __, d, __, k, __, d, __, k, etc.
  
• Ring 3: ai, __, ai__, ai, __, ai, __, ai, etc.
  
• Ring 4: in, n, in, n, in, etc.
  

Ring 1 only has "o" and no longer has "qo".
Ring 2 has "k" and "d", and no longer has "t".
This can give you words like daiin, dain, kaiin, okaiin, etc.

So in this hypothetical example, the effect of the replacement is:

• Words like "qotaiin" suddenly disappear because they only appeared in volvelle 1 which was destroyed.
  
• Words like "okaiin" suddenly appear because they were newly introduced by volvelle 1b and did not exist before.
  
• The net effect of appearances/disappearances make it look as if there's a transformation from "ot-" prefix to "ok-" prefix for the same words before and after.
  
• Words like "odaiin" appear in both volvelle 1 and volvelle 1b, but the frequency increases because it's slightly more likely to arise after the replacement (more "o-" slots in volvelle 1b).
  
• Words generated by volvelle 2 like "chedy" keep appearing at the same frequency, at least until volvelle 2 wears out and gets replaced by a volvelle 2b.
  

If this volvelle replacement keeps happening across the manuscript, the gradual turnover of combos and words would accumulate to the point where most of the "vocabulary" shifts into another vocabulary.

So, the questions I wanted to investigate:

1. Are there systematic transformations happening across the manuscript? (the question you answered here)
   
2. If so, are there multiple transformations happening simultaneously at abrupt points in time?
   
3. If so, can we work backwards from those shifts to reverse-engineer what the volvelles looked like before and after these shifts?
   

If I actually had the capacity to look into those, I would have also published it alongside the justification of why I thought this was the most suitable mechanism in the first place, that is, multiple (around five) simultaneous four-ringed volvelles with alternating blank spaces. As well as a potential cipher that exploits this. Maybe you can do something better with the idea than I can.

[Jorge_Stolfi]

If many dimensions move together, there is a real underlying factor.

The use of the word "move" is already biased towards the interpretation of that "underlying factor" being time.

In the book example, height, width, volume, weight, number of words, number of pages, number of letters per page etc are all correlated variables.  But the ordering of books by any of those attributes is not significant.  It has nothing to do with the order in which they were written, printed, bought, or read -- which seems to be implied if one says that those variables "move together".

You may have seen You are not allowed to view links. Register or Login to view..  It shows that height and width of old manuscripts (at least, those in Rene's dataset) are indeed strongly correlated, so that the points (h,w) fall very close to a straight line.  I see several clusters of points (marked), but Rene apparently sees a continuum of sizes instead.  Either way, while the correlation is significant (and suggests some theories about the Medieval making and marketing of vellum), the ordering of the manuscripts along that line obviously is not significant. 

[Rule reversal] doesn't suggest the ordering [by statistics] is not meaningful. Reversals are exactly what you would expect with changes that are random drifts.

But one would expect the changes to be "random" only if one expects the text to be gibberish.  I guess that this is your preferred theory, yes?  Well, it is not mine. Even before my recent discoveries about the Stars section, my probability for "gibberish" was already close to zero.  So that may be a point where we are bound to permanently disagree.

On the other hand, if the text is meaningful, and one is considering the possibility that the statistical changes may be due to "spelling reforms" or to the Author's word preferences gradually changing due to mere passage of time, one should not expect those changes to be reversed.  They would not be "random".

In fact, even if one subscribes to the "gibebrish" theory, and thus expects the shifts to be random, one should not see several idependent rules being reversed at the same time.

Herbal A and B are the same topic, format, and style, and yet are significantly different. This was the same difference you and I had discussed and was the motivation behind this analysis.

The illustrations strongly indicate that Herbal A and B are about the same general topic (herbalism), and we can see that the text format is basically the same (mostly 1-3 middle-size parags per page).  

But we don't know whether they are in the same style.  Maybe one is in the conditional mood while the other is imperative or descriptive.  Maybe one has harvesting or dosage information that is lacking in the other.  Maybe one notes the effects of each plant on life extension while the other seems more preoccupied with digestive problems, or with health problems specific to women.  

Anyway, the massive difference in word usage between those two sets of pages, compared to the relative similarity between pages within each set, makes it necessary to treat Herbal-A and Herbal-B  as separate sections.  Whereas the additional proposed splits (like Rene's) do not seem to be sufficiently justified.

Even if you see differences between (say) "early Herbal-A" and "late Herbal-A", they may have explanations that do not justify treating those sub-sets separately.

For example, a certain old book (whose name shall not be uttered here) has a large number of short parags, each describing a remedy and its applications or benefits.  In the oldest copies of that book, the remedies are grouped into three major sections by toxicity.  First come the "High grade" remedies that are essentially tonics that can be taken for long periods without harmful effects.  Then there are "Medium grade" remedies that should be taken only to cure specific problems and only until the problems are resolved.  Then there are the "Low grade" ones which are quite toxic and should be taken only as a last resort, in the smallest possible doses and under close surveillance by the physician.

The entries have all similar format, language, and semantic structure, and style.  But most "High grade" ones have the words "long term use", and they list mostly common benefits like long life, strength, intelligence, good complexion, virility, milk production, fertility, etc.  The Medium grade entries hardly have that "long term use", except to warn about side effects like hallucinations or change of mood.  On the other hand they list many more diseases that cannot be treated with the tonics of the first section.  Then the Low grade entries list more serious conditions and may have extra warnings about toxicity and side effects.

However, other copies of that same book have the entries sorted instead by nature -- Mineral, Vegetable, and Animal.  And each of these sub-sections will have its own set of specific words, like "root", "milk", etc.  

And maybe there are copies out there in still another order -- like alphabetic, by governing planet, by geographic origin, whatever. 

Anyway, the statistical differences between those sub-sections seem to be small, involving maybe 50-100 words out of the 1000 lexemes used in the book.  They are dwarfed by the word usage differences between that book and any other book, even on the same topic.  

The point of this example is that splitting that book into sub-sections based on word frequencies (or the character and slot pair frequencies that are determined by them) would be more confusing than illuminating.  If the book was in code or an unknown language, such splitting would not help deciphering it; it would instead distract from other more effective analyses.   

Which I think is the case for VMS sections like Herbal-A or Stars: at the present stage, I do not see the justification for splitting them into sub-sections.

Incidentally, the example of that book also brings up another important point: when comparing the VMS with other "control" books, one should try to use books that are likely to have similar style and text structure (visible and semantic).  Thus one should compare Herbal-A to other herbals like the You are not allowed to view links. Register or Login to view., rather than to the Vulgate or to Oresme's treatise.

All the best, --stolfi