Voynich words evolve over a linear sequence

[npcompl33t]

For the last several weeks, I have been working on an attempt to 'translate' Herbal A to Herbal B. This turned up many probable transformations that turn words from Herbal A into Herbal B, preserving frequencies, but many of these transformations only worked between two sections, never collectively across the entire manuscript.

You are not allowed to view links. Register or Login to view., I laid out a hypothesis and evidence for a Voynich chronology, showing that Voynich word construction shows drift that is equivalent to stepwise evolutions, that sometimes undo previous results. The methods I used are identical to those used to reconstruct the chronology of Egyptian tombs, by modeling changing pot styles. 

If i apply this to my rule/transformation work, and instead of modeling rules that must be true across the entire manuscript, model them as stepwise transitions between each stage, everything falls into place. Frequencies all mostly align between sections, and once large gaps become a series of gradual transitions.

Note that the transformations below are not the result of manually linking individual words. I define a series of transformations that must apply to all words. For example, *ol -> *eol , then *eol-> *edy. 

A small set of these rules, about 20 or so, can explain most frequency differences between words across the entire manuscript, when applied in sequence, similar to how linguistic reconstruction models phoneme changes between words.

You are not allowed to view links. Register or Login to view.

At any given step, these rules only capture a small amount of volume, with most words retained. 

   

However, if we actually see how many words remain consistent from start to end, we see an obvious gradual turnover of words. Each step changes a small amount, but over several steps these collectively add up, eventually resulting in large turnover across multiple steps. The chart below shows how many words from each step are retained into subsequent steps. Notice the gradual decline each step. By the end of the sequence, very little is directly retained from 1a.

   


Worked Examples

The chart below shows the evolution of the word 'chol'. Looking at just Currier A/ Currier B, there appears to be a dramatic gap, with chol appearing to mostly vanish from voynich vocabulary. However, in the chart below, we can actually see chol gradually declining and shifting into other words, not a single large jump. Chols transition into chedy is actually gradual, with its roots in the 1.5 'transitional A' section when it first divides into cheol. 

   


Below shows chor. This also appears like a dramatic jump between Currier A/B, but we can now see that its decline is actually gradual, slowly shifting into competing forms at each step, before finally vanishing in Stars.

   


Shol, another example.

   



Note these changes are sometimes undone, for example chol appears to be mostly turned into chedy in 3b, before reverting partially back into chol in 3c. This is consistent with what you would expect from random drift, some changes end up 'undoing' themselves. 


The missing sections, and Currier A/B

I strongly suspect that some of the most extreme differences between Currier A/B, like the appearance of the 'ed' bigram, would exhibit gradual changes in the missing bifolia. Already in the above charts, we can see that many of the A/B differences appear to be the results of transformations that began in the 1.5a transitional herbal section, and i expect that specifically missing quires P, R, and H, (f91, f92, f97, f98, f59-f65) would all show a more gradual transition, with 1 of either quire P/R occurring before pharma, and H and the other occurring afterwards. F58 and the cosmo/astro prose collectively represent a small amount of text, but we can already begin to see some of this occurring there.


The question of semantic meaning vs self citation
The above rules do not settle the question of semantic meaning vs self citation. If this linear evolution represents a gradual turnover, with scribes mostly copying existing words, i think it would be unusual for some forms to actually have their frequency mass preserved across the entire chain like this.

For example, if words truly lacked semantic meaning, there would be no reason why chol could not just simply gradually vanish with no replacement. That does not seem to be what is occurring here. Instead, words frequencies are mostly preserved downstream. It's possible some form of self-citation could also do this, but it would have to be fairly complex to preserve information in this way, and in my opinion, at least currently I think this actually points more to an evolution of the script or writing system.

In summary, I don't think there is enough evidence to fully resolve this question at present.

The exact rule changes

The rule changes i am using are all You are not allowed to view links. Register or Login to view., in the rule manager tab. I do not claim this is the full comprehensive set, just that the current set appears to capture most of the drift under a fairly small set of rules. I have no doubt they could be improved. I suspect applying linguistic reconstruction techniques would be a valuable approach. You can actually create and test your own rules on the site.

[JoJo_Jost]

@ You are not allowed to view links. Register or Login to view.

Nice work!

---
As I've said before, if you look at the structure as a table, you can clearly see that this table structure “evolves.” And that would be typical of a cipher that evolves as it is written—in this case, additional table cells are added because the existing ones are obviously insufficient. Chol’s problem, on the other hand, shows that there was an issue there—perhaps with the back-translation—it was adjusted, and then resurfaced, likely to generate further variety.

So I see it similarly: there is a development. 

[And I could even imagine that the entire VMS is a kind of feasibility study for this cipher—and that it consists of several forms that were simply recorded on poor-quality vellum during its development. But of course, that’s just another unverifiable theory .]

[nablator]

And that would be typical of a cipher that evolves as it is written—in this case, additional table cells are added because the existing ones are obviously insufficient

But Q13 is more degenerated (devolved) than evolved: less lexical diversity (smallest MATTR).

[JoJo_Jost]

Aber Q13 ist degenerierter als weiterentwickelt: weniger Vielfalt, kleinstes MATTR.

Yes, I didn’t explain that correct. Q13, with its extremely high number of “y” spaces, is a special case—which is also one of the reasons why I think they were experimenting here.

For example (I hope I’m allowed to briefly mention this as an example in this context), if these bigrams separated by spaces were supposed to be vowels, then they were encoded using homophones. And in q13 no other variant was chosen - theoretical - so it looks like an experiment...

But even some of the multiple word repetitions (at least in my approach to solving it) seem to be feasibility experiments, even though they could have been used to encrypt entire sentences.

The difference i mention is most clearly seen between Herbal A and B.

Labels behave differently again.

And “Stars” is different again, too — which gives me the impression that it’s the most refined form.

But of course, these are all just “conjectures.” 

[MarcoP]

Thank you npcompl33t, I find the subject very interesting!

In my opinion, there's much wasted space in the images in post #1: if empty horizontal space could be removed, maybe the graphs would be easier to read.
Also, I would prefer to see rules listed and discussed in the forum post, rather than at an external page that I expect will change in the future.

if words truly lacked semantic meaning, there would be no reason why chol could not just simply gradually vanish with no replacement.

I am not sure I understand what is being said here. "replacement" is a theory, one can see the same phenomenon as “gradually vanishing”. It seems you are drawing an inference from an assumption that may or may not be correct.

Also, I am not sure the inference is correct by itself. Why cannot meaningless words gradually turn into different but similar meaningless words? Gaskell and Bowern found that spontaneous written gibberish has highly variable features: not much can be excluded a priori. In particular they found that, when compared with ordinary text, Voynichese “adjacent words are [less] different from one another relative to the average” (very low wordchange_mean), and this can also happen in spontaneous gibberish. This small-scale observation may be related with the greater shifts we observe on longer parts of the text. The samples they checked are too small to observe a substantial drift, but I don’t think we can safely assume that it wouldn’t happen.

EDIT: Maybe a detailed discussion of a limited set of words, like You are not allowed to view links. Register or Login to view., would also make the theory easier to understand.

[Mauro]

I think you probably found the most stringent constraints any viable Voynich theory must respect (btw, I really hope for a peer-reviewed article). Great job!

So, I really hope your results hold (I think so, but I'm no expert in statistics). Let me sum them up, as a check of my own understanding.

• You confirm the basic unit of the VMS is the bifolio (1).
  
• You found and retrieved a gradient in the distributions of bigrams in each bifolio (2) (3)
  
• And now you have determined a rather small set of textual 'transition rules' which smoothly transform the bigrams distribution of a bifolio into the next (you actually said 'the words' in this thread but I though you were using bigrams P(b|a) distributions? can you clarify this to me please?). Which, taken face value, is a great result.
  

I'd really like to know more about the 'transition ruels' and as @Marco P, I'd really appreciate a post here summing the transition rules (and their statistical relevance). I'm sorry but digging in a complex website to get the informations out is not so easy (4)


(1) As originally proposed by @LisaFaginDavis using a different approach.
(2) The existance of various kinds of 'gradients' in many different forms has been proposed by many over the years but, to my knowledge, always considering sections of the manuscript, never single bifolios, and noone succeeded before in getting a full picture.
(3) Which allowed you to put the manuscript in a, let's say, chronological (or reverse-chronological) order. Lisa Fagin Davis did the same using different mathematical tools and it would be very interesting to read about a comparison of the two results (I got little time for Voynich at the moment   )
(4) I'm sure it's all perfectly clear to you but it would take a lot of time to go through. Go figure, even the softwares I myself wrote just a few years  ago make me scratch my head if I happen to re-open them 

[npcompl33t]

I think you probably found the most stringent constraints any viable Voynich theory must respect (btw, I really hope for a peer-reviewed article). Great job!

So, I really hope your results hold (I think so, but I'm no expert in statistics). Let me sum them up, as a check of my own understanding.

• You confirm the basic unit of the VMS is the bifolio (1).
  
• You found and retrieved a gradient in the distributions of bigrams in each bifolio (2) (3)
  
• And now you have determined a rather small set of textual 'transition rules' which smoothly transform the bigrams distribution of a bifolio into the next (you actually said 'the words' in this thread but I though you were using bigrams P(b|a) distributions? can you clarify this to me please?). Which, taken face value, is a great result.
  

I'd really like to know more about the 'transition rules' and as @Marco P, I'd really appreciate a post here summing the transition rules (and their statistical relevance). I'm sorry but digging in a complex website to get the informations out is not so easy (4)

Yes, this analysis is entirely different from the P(b|a) distributions. I basically took the sequence that analysis uncovered, then started looking at actual words to attempt understand how the abstract bigram changes manifest at the word level. This shows that word level turnover aligns with the chronology analysis, and it presents a set of rules that explain much of the shift among common words.

However, I don't want to overstate the 'transition rules'. I am not claiming my rules smoothly translate all words across all steps. I have found evidence that a small set of rules can cover the changes across most of the more frequent words, but they do not yet fully explain every shift in the bigram distribution. My work up until now required matching words between sections based on frequency which is only possible with common words.

I am now trying to expand my work to be able to cover rare word and hapax transformations. There does appear to be some differences in how these changes affect common vs rare words. I think this is compatible with both self-citation and semantic preserving drift, resolving that is going to take more work.

As a community, I think the most pressing things to resolve are:

• A more comprehensive set of word transformations
  
• A better understanding of these changes across each 'cluster' and bifolio, especially the bifolios in the middle of the sequence as the ordering there is more stable than the end sections
  
• how the transformations propagate between rare and common words
  
• what words appear to be affected by what changes
  
• whether the transformations appear to preserve information or semantic meaning
  

I am working on expanding this work to dig into these but I am not claiming to have already answered all of the above questions.

[npcompl33t]

Thank you npcompl33t, I find the subject very interesting!

In my opinion, there's much wasted space in the images in post #1: if empty horizontal space could be removed, maybe the graphs would be easier to read.

The horizontal space there is relevant, they are spaced proportionally to the identified time sequence in the previous post. So it shows the approximate relative time between each section.  I can post them with equal separation if that would help, its an easy toggle already built into the app.

Also, I would prefer to see rules listed and discussed in the forum post, rather than at an external page that I expect will change in the future.

This is fair, I want to be clear I am not claiming this rule-set is comprehensive or entirely correct. I think it shows that certain specific transformations can move the frequency distribution, but I am still researching and evaluating the rules and I'm not prepared to defend them all individually yet. The intention here was to show that this is possible and a worthwhile avenue for research, not to claim these rules are the canonical set. If i was making the later claim, i would absolutely put them here on the forum. I can definitely post a list of which transformations i am using above if you think that is helpful, i just want to be clear that I am not arguing they are comprehensive.

I am not sure I understand what is being said here. "replacement" is a theory, one can see the same phenomenon as “gradually vanishing”. It seems you are drawing an inference from an assumption that may or may not be correct.

Your point strikes at the core of the matter, which is I don't think this has been adequately discussed. All previous debate has focused on the existence of the gradient, not on what types of features might distinguish gibberish self copying vs some type of semantic information presrerving transformation,. 

My point was more that both semantic preserving drift and meaningless copying should manifest differently. If the text encodes semantic meaning, then any transformations ultimately need to preserve information. I would greatly appreciate a discussion around this.

If chol were to vanish without a candidate replacement occurring at roughly the same frequency, that would be strong evidence for self citation or gibberish. Chol ultimately means nothing, so the scribes aren't concerned with making sure its descendants occur at roughly the same rate. 

The other explanation would be Lexical Replacement, where one word gradually replaces another. It happens all the time in real languages. These transformations appear to preserve frequencies in the above example, which is not necessary for self-citation but is to preserve semantic meaning. 

EDIT: Maybe a detailed discussion of a limited set of words, like You are not allowed to view links. Register or Login to view., would also make the theory easier to understand.

i think getting into the specifics is going to be critical to resolve this. In torstens theory, he has both cheo and chey as a part of chol's transformation. I do not think that makes sense. It makes more sense , in my opinion for chy -> chey. They appear to mostly preserve frequency mass when calculated this way. The difference here is extremely important, because failure to preserve information or losing words outright in my opinion would be strong evidence against semantic meaning. 

More broadly, i think collectively discussing what sorts of markers would help distinguish semantic meaning vs self citation within the context of the gradient would be extremely valuable.

[npcompl33t]

Here is the list of full transformation rules i am using in the above charts. 

Like i mentioned, I am not 100% sure on all of them and have indicated my relative confidence in them using 3 tiers, hypothesis, theory, and rule. With rule being the most confident and hypothesis being the least.

   

[Jorge_Stolfi]

I am still trying to parse all the data you have produced.  

Who or what is "Golf"?  The Voynich Search Engine does not seem to know either...

My first impression is that there may be a bit of circularity in your approach.  You separate the pages into groups based on certain text statistics, sort the groups into an order from "Early A" to "Bio" according to some of those statistics, and then observe that there is a gradation in some other text statistics along the same ordering.  

This observation only shows that there is some correlation between the criteria used for sorting the sections in the first place and the other text statistics.  Such correlations are expected, but they do not confirm that the ordering is significant.

Going back to the book sorting example, if you sort a hundred random books by height, you will find that the width also seems to gradually vary as you move along that sequence.  That observation shows that height and width are correlated variables, which is a significant observation; but it does not make the ordering any more meaningful.

The fact that some of the word replacement rules are reversed at some point suggests that the ordering in fact is not meaningful. 

Here are two "null hypotheses" that would be the alternative to the "linear evolution" claim:

• H0. "The variations in the text statistics from section to section are essentially all determined by changes of topic, text format, text structure, and style."
  
• H1. "The pages within each section are statistically homogeneous, and can be well modeled as random Gaussian-like deviations from a single central word frequency distribution."
  

Hypothesis H0 says, in particular, that Stars and Bio have similar word frequency distributions (WFDs) because they deal with the same topic, and/or possibly because they use the same style (like imperative vs.conditional, telegraphic vs. full sentences, etc.)  But that does not mean that they were created in sequence.  It could well be that Bio was the first section to be written and Stars the last one, or the other way around; with all the other sections written in between.

(In particular, I think I know what the word chedy means in the Stars section; and it is something that would naturally be heavily mentioned in Bio too, but not so much in the other sections.)

Hypothesis H0 does not deny that some changes in the WDFs may be directly caused by the mere passage of time, which would have caused the Author to gradually change his word usage preferences.  It says only that such changes are swamped by the changes due to topic &c.  

H0 also does not deny that the Author may have made a massive "spelling reform" at some point; but implies that a spelling change would be hard to distinguish from a change of topic that required a substantial change in the lexicon.

So, what arguments or evidence do you have against H0 and H1?

All the best, --stolfi