The Voynich Ninja

(12-06-2019, 10:46 PM)nablator Wrote: You are not allowed to view links. Register or Login to view.If I am not mistaken, when the vertical distances are restricted to the same page the diagrams are much different:

The purpose of figure 4 is to illustrate the level of context dependency within the VMS. What we actually say about the context dependency is: "No obvious rule can be deduced which words form the top-frequency tokens at a specific location, since a token dominating one page might be rare or missing on the next one. However, all pages containing at least some lines of text do have in common that pairs of frequently used words with high mutual similarity appear" (Timm & Schinner 2019, p. 3). 

Figure 4 is indeed not perfect since it only gives a high level view for the whole VMS. It is an improvement to compare figure 4 with a corresponding graph showing the results restricted to the same page. The result confirms our statement that each page is dominated by words with high mutual similarity and that tokens dominating one page might be rare or missing on the next one (see also You are not allowed to view links. Register or Login to view., p. 14 and You are not allowed to view links. Register or Login to view., p. 3f). This result is also confirmed if we look into individual pages (see 1.3 You are not allowed to view links. Register or Login to view.). This result confirms that in the VMS tokens co-occur with similar ones. This observation also confirms that local repetition is a key feature for the VMS.

Anyway, the whole discussion about an alternative experiment to simulate a high level graph as given in figure 4 is at best a sidetrack. Moreover, an alternative experiment suggests that you accept that the original experiment indeed confirms that it was possible for a medieval scribe to generate meaningless text which statistically appear to have features of language. Therefore I would suggest that we focus the discussion to the observations and the argumentation presented in our paper (see for instance my posts about the You are not allowed to view links. Register or Login to view. presented in chapter 2 and about the You are not allowed to view links. Register or Login to view. presented in chapter 5).

Torsten,

I have no reason to doubt your statistics so I trust they are correct.

I do find the auto-copying hypothesis not sufficient as a method for constructing the Voynich MS text. Not every word in the MS is a copy with edit distance 0 or 1 from a recent word, for any reasonable definition of 'recent'. The percentage of such words for different definitions of 'recent' could be computed but this has not been done. What if more than 50% of words is not a copy with edit distance 0 or 1 from within the last 10 lines? Then more than half the text is not explained.


My biggest problem is with the conclusion. It does not follow that the correlation between edit distance and distance in the text is evidence that the text is meaningless.

Quote:Anyway, the whole discussion about an alternative experiment to simulate a high level graph as given in figure 4 is at best a sidetrack.

This is not a sidetrack. It is an example how a meaningful text can also exhibit this unusual property, and demonstrates that the conclusion is not valid.

It is of course perfectly fine to be of the opinion that the text is meaningless, but this does not follow from the evidence presented.

(15-06-2019, 06:29 AM)ReneZ Wrote: You are not allowed to view links. Register or Login to view.Torsten,

I have no reason to doubt your statistics so I trust they are correct.

René,

so you agree at least to the You are not allowed to view links. Register or Login to view. described in chapter 2 "Context-dependent self-similarity" (You are not allowed to view links. Register or Login to view., p. 2ff). Thanks for this clarification. 

(15-06-2019, 06:29 AM)ReneZ Wrote: You are not allowed to view links. Register or Login to view.I do find the auto-copying hypothesis not sufficient as a method for constructing the Voynich MS text. Not every word in the MS is a copy with edit distance 0 or 1 from a recent word, for any reasonable definition of 'recent'. The percentage of such words for different definitions of 'recent' could be computed but this has not been done. What if more than 50% of words is not a copy with edit distance 0 or 1 from within the last 10 lines? Then more than half the text is not explained.

That every word is a copy with edit distance 0 or 1 from a recent word is a You are not allowed to view links. Register or Login to view. argument. In fact most times the scribe made at least two modifications (see You are not allowed to view links. Register or Login to view., p. 14ff). Moreover, modification rule 3 is to combine two source words. If the scribe used <ol> and <chedy> to generate <olchedy> the edit distance between <ol> and <olchedy> is obviously much higher than 1.

You falsely suggest that the repetition must be obvious like on page You are not allowed to view links. Register or Login to view. (see You are not allowed to view links. Register or Login to view.) or on page You are not allowed to view links. Register or Login to view. (see You are not allowed to view links. Register or Login to view.). Such a suggestion doesn't make any sense. Keep in mind that the scribe was seeing what he was writing. If the repetition was obvious in his eyes he had complete freedom to vary some details of the generating algorithm on the spur of a moment. For instance he was for sure able to add a word like <daiin> out of his head, or he could start a new paragraph, or he could invent a new glyph like You are not allowed to view links. Register or Login to view., ... 

We have discussed this subject already in 2017. Back then you wrote: "However, the objection ... does not apply in this case. Changes can be made arbitrarily, at any point" (see You are not allowed to view links. Register or Login to view.).

(15-06-2019, 06:29 AM)ReneZ Wrote: You are not allowed to view links. Register or Login to view.My biggest problem is with the conclusion. It does not follow that the correlation between edit distance and distance in the text is evidence that the text is meaningless.

Indeed, the correlation between edit distance and distance in the text alone is not enough to demonstrate that the text is meaningless. Where did you read such a statement?

(15-06-2019, 06:29 AM)ReneZ Wrote: You are not allowed to view links. Register or Login to view.

Quote:Anyway, the whole discussion about an alternative experiment to simulate a high level graph as given in figure 4 is at best a sidetrack.

This is not a sidetrack. It is an example how a meaningful text can also exhibit this unusual property, and demonstrates that the conclusion is not valid.

There is no contradiction between your experiment and my conclusion. My conclusion from this property was "A feature of the VMS is that similarly spelled glyph groups are used together on the same pages near to each other. This means, the reason why similarly written words have similar frequencies is that they appear together on the same pages" (You are not allowed to view links. Register or Login to view., p. 14). It is important that "The respective frequency counts confirm the general principle: high-frequency tokens also tend to have high numbers of similar words" (You are not allowed to view links. Register or Login to view., p. 6). Because of both observations I conclude that "the scribe was writing similarly spelled tokens near to each other because they depend in some way on each other" (You are not allowed to view links. Register or Login to view., p. 14).

Your experiment doesn't include the observation that similarly written words occur with similar frequencies. But it was still necessary to you to generate a number of code words during writing. Only this way it was possible to simulate the high level of context-dependency for the VMS. In this way your experiment confirms that some kind of relation between similarly spelled words exists.

(15-06-2019, 06:29 AM)ReneZ Wrote: You are not allowed to view links. Register or Login to view.It is of course perfectly fine to be of the opinion that the text is meaningless, but this does not follow from the evidence presented.

Shouldn't you attack the argumentation for a meaningless text in this case? Our argumentation for a meaningless text is: "the high regularities of the VMS text significantly limit the maximal amount of information possibly hidden within the 'container', virtually rendering it useless" (You are not allowed to view links. Register or Login to view., p. 17).

If you want to argue that the VMS contains a message you have to answer the following question: Where is the information hidden? 

The shape of each glyph corresponds to the previous glyph. The tokens respond to previously written tokens on the same page. The line works as a functional unit. The text on each page responds to the page layout. How much information could you hide in a text that follows all this different rules?

You could still argue that maybe it is necessary to count the occurrences of a certain glyph in each line or that only a glyph in a certain position in a line or paragraph contains meaning. But this would mean that it was only possible to hide at best one letter in a line of text.

Quote:For instance he was for sure able to add a word like <daiin> out of his head, or he could start a new paragraph, or he could invent a new glyph

This is exactly a point I was asking about much earlier on. You can't have auto-copying without some initialisation, and this part is not addressed. The text clearly shows that this was not just a one-off to start the process (like only the first word in the MS) but must have been going on all the time, and relatively frequently.

(17-06-2019, 05:26 AM)ReneZ Wrote: You are not allowed to view links. Register or Login to view.

Quote:For instance he was for sure able to add a word like <daiin> out of his head, or he could start a new paragraph, or he could invent a new glyph

This is exactly a point I was asking about much earlier on. You can't have auto-copying without some initialisation, and this part is not addressed. The text clearly shows that this was not just a one-off to start the process (like only the first word in the MS) but must have been going on all the time, and relatively frequently.

Where did you ask something about "initialization"? What needed "initialization" in your opinion? It is not clear what you mean.

There was only the need to initialize the first line on an empty page. I have addressed this issue You are not allowed to view links. Register or Login to view.: See Timm & Schinner 2019, p. 10: "There was a similar problem for the author of the VMS every time he/she was starting a new (empty) page. ...". See also Timm 2015, p. 30ff: "To use the described text generation method for the initial line of a page it was necessary to use another page as a source. ...".

But you probably mean something like the initialization of the network of similar words or the initialization of the algorithm. I have addressed this issue You are not allowed to view links. Register or Login to view., You are not allowed to view links. Register or Login to view., and You are not allowed to view links. Register or Login to view.. 

People usually do not plan to think about something in advance or build a complete decision matrix. They plan some goal and try to reach it. This is what people usually do if they want to try to write some kind of text and this is what the scribe was doing while writing the VMS: "Keep in mind that the VMS was created by a human writer who had complete freedom to vary some details of the generating algorithm on the spur of a moment" (Timm & Schinner 2019, p.16). 

There was for instance no plan to write <daiin> seven times and <dain> six times on page f1r. The scribe was not counting the words he was writing (see Timm 2015, p. 7). <daiin> and <dain> occur seven and six times on page You are not allowed to view links. Register or Login to view. since every decision to copy a particular word has some influence to the text. This is the case since every word is at the same time the result of the copying process and it is also a possible source word for further copying steps (see You are not allowed to view links. Register or Login to view.). Since the words are copied from each other it is necessary that a number of similar words are used more frequently than others. But it is not necessary that this words must be <daiin> and <dain>. This is the reason that words with many similar ones are frequently used and words with no similar ones occur only once. This is the explanation for the observation that no "obvious rule can be deduced which words form the top-frequency tokens at a specific location because a token dominating one page might be rare or missing on the next one." (Timm & Schinner 2019, p. 3) and since no unchangeable decision matrix was used it is necessary that the text was changing over time (see Timm 2015, p. 25). Only therefore we see the gradual evolution of a single system from Currier A to Currier B (see Timm & Schinner 2019, p. 7).

(23-06-2019, 06:49 PM)MarcoP Wrote: You are not allowed to view links. Register or Login to view.I am quite sure that a procedural method can reproduce any statistical feature of Voynichese (or of any other text).

The self-citation method can only reproduce statistical features similar to Voynichese (see Timm & Schinner 2019, p. 15f).

(23-06-2019, 06:49 PM)MarcoP Wrote: You are not allowed to view links. Register or Login to view.I don't understand why self-citation should cause the initial glyph of one word to depend on the final glyph of the previous word.

The previous word is the last written word. To copy an element of the previous word is therefore an obvious choice. The only problem is that this choice is indeed obvious. The shape of <q> is quite similar to the shape of <y>. See for instance the difference between <y> and <q> in <yoky> and <qoty> on page You are not allowed to view links. Register or Login to view..

It is possible to demonstrate that indeed an element of the previous word was copied. Consecutive tokens start more often than expected with the same initial glyphs or end with the same final glyphs. See table 1. "Initial-Initial dependency" and table 3 "Final-Final dependency" in the Analysis of word dependency by Vladimir Sazonov: "As it is clear from the tables 1 and 3, the words very often reproduce the same form (the same initial or the same final) in the next word. For example, -aiin or qo- or -dy can repeat many times with good density" (You are not allowed to view links. Register or Login to view.).

(23-06-2019, 06:49 PM)MarcoP Wrote: You are not allowed to view links. Register or Login to view.Also, please correct me if I am wrong, but I think the generated text here doesn't feature any clear dependency between the last glyph of a word and the first glyph of the next word?

"Keep in mind that the VMS was created by a human writer who had complete freedom to vary some details of the generating algorithm on the spur of a moment. ..." (Timm & Schinner 2019, p. 16). With other words we didn't argue that the text was created by a computer program and we didn't argue that our program is able to simulate the complexity of a human mind.

Beside line breaks numerous similar rules for describing some kind of local repetition exists. Even if I would try to add each rule it would still be possible to point to an another similar rule. At the same time each additional rule would increase the complexity of the program and it would be argued that the program is too sophisticated. Therefore our goal was to keep the algorithm as simple as possible.

(23-06-2019, 06:49 PM)MarcoP Wrote: You are not allowed to view links. Register or Login to view.Something I particularly like about your approach is that you take reduplication and quasi-reduplication seriously. You have created a system that nicely models them.

I didn't create the self-citation method. In my paper from 2015 I try to reverse engineer the way the text of the VMS was written. See chapter 6 "Evidence" (Timm 2015, p. 12f) and chapter 7 "The text generation method" (Timm 2015, p. 14f). This means the self-citation method is a way to describe Voynichese (see also Timm & Schinner 2019, p. 9: "We start with focusing on the set of rules most likely used in the VMS. ...").

With other words the self-citation method is the most simple way to explain the observed statistical features. Since every word is at the same time the result of the copying process and also source for generating new words it is necessary that every Voynich-word is n-th copy of any other Voynich-word. The only possible outcome is therefore a network of similar words. 

One result of the self-citation method is that a more frequently generated word is copied more frequently. This means that a frequently generated word results in many similar words whereas each similar token also increases the chance that the original word is generated again.

This recursive process results in the observation that "high-frequency tokens also tend to have high numbers of similar words" (Timm & Schinner 2019, p. 6) and explains that similar tokens occur with similar frequencies. It is therefore necessary that a text generated by the self-citation method fulfills both of Zipf’s laws. 

Since similar words have similar length it is also necessary that the network of similar words results in a "binomial-like" word length distribution.

Because it is handy to copy a word from the same position some lines above (see Timm 2015, p. 18) the self-citation method results in long-range correlations like the correlations found for the Voynich-text (see Timm & Schinner 2019, p. 14).

(25-06-2019, 12:00 AM)Torsten Wrote: You are not allowed to view links. Register or Login to view.

(23-06-2019, 06:49 PM)MarcoP Wrote: You are not allowed to view links. Register or Login to view.I don't understand why self-citation should cause the initial glyph of one word to depend on the final glyph of the previous word.

The previous word is the last written word. To copy an element of the previous word is therefore an obvious choice. The only problem is that this choice is indeed obvious. The shape of <q> is quite similar to the shape of <y>. See for instance the difference between <y> and <q> in <yoky> and <qoty> on page f79r.

It is possible to demonstrate that indeed an element of the previous word was copied. Consecutive tokens start more often than expected with the same initial glyphs or end with the same final glyphs. See table 1. "Initial-Initial dependency" and table 3 "Final-Final dependency" in the Analysis of word dependency by Vladimir Sazonov: "As it is clear from the tables 1 and 3, the words very often reproduce the same form (the same initial or the same final) in the next word. For example, -aiin or qo- or -dy can repeat many times with good density" (V. Sazonov 2003).

Thank you, Torsten.
I will first comment your second argument. I interpret it as a suggestion that initial-initial repetition and/or final-final repetition can cause a bias in the distribution of word-break combinations. While in principle this is true (i.e. a sufficient amount of repetition can cause considerable shifts in word-break combinations) it is not what happens in the VMS.
In the case of -y.q- only a subset of initial-initial repetitions can cause this word break combination: q*y.q, i.e. a word starting with q- and ending with -y, followed by a second word starting with q-.
If one considers the whole manuscript, about 2000 occurrences of -y.q- would be expected, while about 3500 actually occur. The difference between the two (1500) is the number of unexpected occurrences. This graph shows how different X*y.q- sequences contribute to the unexpected combinations (I only included the most frequent "prefixes").

[attachment=3055]

As you can see, q*y accounts for about 400 of the unexpected occurrences: they are many, but q- repetition does not nearly explain the observations. The other 1100 unexpected occurrences are due to other prefixes. The graph also shows that what "triggers" the presence of q- is -y alone: the preference is quite uniform independent of the prefix of the previous word.

Turning to your first argument, if I understand correctly what you write, it appears to be more in agreement with the observations: the word after -y is modified or selected in such a way that it starts with q-.
I understand your statement that "The shape of <q> is quite similar to the shape of <y>" as your explanation for why q- appears to be systematically chosen to be written immediately after -y. 
Graphical similarity is somehow subjective. One can also say that <y> looks like <o>, but the word-break combination -y.o- is slightly dis-preferred (i.e. occurs less often than expected).

Similarly, -o.y- is dis-preferred. Moreover, all the immediate repetitions of the same character across words are dis-preferred (in various degrees): if the idea were having similar characters across word-breaks, -y.y- could have been an option.

If one examines word-break preferences in their entirety, it is possible to point out a different explanation that applies to a large part of the observations. In word-break combinations, both -o and -y prefer to be followed by the gallows characters and <d> <s> <r> <l>. These same preferences can be observed for combinations "inside" words. Words containing the sequence "yo" are rarer than expected, while words containing "ok", "od", "yk", "yd" etc are more frequent than expected. With the exception of <e> and <i>, characters tend not to occur consecutively inside words, as it happens in word-break combinations.

In general, the preferred combinations do not seem to encourage the consecutive occurrence of graphically similar characters. E.g. o/y are different from the gallows and from r/s. The general principle could possibly be the reverse: similar characters do not typically follow each other.

What can be said is that the preferences in word-break combinations largely reflect the preferences for consecutive characters inside words.

But of course this is not a hard rule, there are exceptions and various problems. Still this seems to work much better than assuming that word-breaks combinations favour the presence of similar characters across word-breaks.

(25-06-2019, 12:00 AM)Torsten Wrote: You are not allowed to view links. Register or Login to view.

(23-06-2019, 06:49 PM)MarcoP Wrote: You are not allowed to view links. Register or Login to view.Also, please correct me if I am wrong, but I think the generated text here doesn't feature any clear dependency between the last glyph of a word and the first glyph of the next word?

"Keep in mind that the VMS was created by a human writer who had complete freedom to vary some details of the generating algorithm on the spur of a moment. ..." (Timm & Schinner 2019, p. 16). With other words we didn't argue that the text was created by a computer program and we didn't argue that our program is able to simulate the complexity of a human mind.

Beside line breaks numerous similar rules for describing some kind of local repetition exists. Even if I would try to add each rule it would still be possible to point to an another similar rule. At the same time each additional rule would increase the complexity of the program and it would be argued that the program is too sophisticated. Therefore our goal was to keep the algorithm as simple as possible.

Thank you for clarifying!
Since in the paper you state that your algorithm "is able to reproduce all the intriguing statistical key properties of the original VMS", I was unsure if you considered the unexpected word-break combinations a key property of Voynichese or not. You made things clear now. I also understand your reasons for not modelling a larger subset of the known properties of Voynichese.

(27-06-2019, 02:11 PM)MarcoP Wrote: You are not allowed to view links. Register or Login to view.I will first comment your second argument. I interpret it as a suggestion that initial-initial repetition and/or final-final repetition can cause a bias in the distribution of word-break combinations. 

My point is that initial-initial repetition, final-final repetition, and word-break combinations have the same cause. The cause is that words co-occur with similar ones. This means that word-break combinations are just another samples of local repetition.

(27-06-2019, 02:11 PM)MarcoP Wrote: You are not allowed to view links. Register or Login to view.Turning to your first argument, if I understand correctly what you write, it appears to be more in agreement with the observations: the word after -y is modified or selected in such a way that it starts with q-. I understand your statement that "The shape of <q> is quite similar to the shape of <y>" as your explanation for why q- appears to be systematically chosen to be written immediately after -y. Graphical similarity is somehow subjective. One can also say that <y> looks like <o>, but the word-break combination -y.o- is slightly dis-preferred (i.e. occurs less often than expected).

Similarly, -o.y- is dis-preferred. Moreover, all the immediate repetitions of the same character across words are dis-preferred (in various degrees): if the idea were having similar characters across word-breaks, -y.y- could have been an option.

A general principle of the self-citation method is to copy glyph sequences and to avoid obvious repetitions by replacing glyphs with similar ones (see Timm 2015, p. 5). To repeat the same glyph, token, or modification rule too often would result in obvious repetitions and would therefore violate this general principle. Moreover as demonstrated in your paper the distribution of word-break combinations is not consistent for the whole VMS. This suggests that we speak about rules that were developed and/or modified during writing the VMS. I didn't see any need to apply the same modification rule for every glyph. In fact it is possible that the scribe developed individual rules for certain glyphs and it is also possible that this rules gets modified within the text.

(27-06-2019, 02:11 PM)MarcoP Wrote: You are not allowed to view links. Register or Login to view.If one examines word-break preferences in their entirety, it is possible to point out a different explanation that applies to a large part of the observations.

Our goal was to examine the key properties for the whole VMS-text (see chapter 2 in Timm & Schinner 2019, p. 2ff). Word-break preference are just one element within the whole VMS-text. In my eyes it is therefore not helpful to focus solely on word-break preferences.

(28-06-2019, 02:43 PM)Torsten Wrote: You are not allowed to view links. Register or Login to view.My point is that initial-initial repetition, final-final repetition, and word-break combinations have the same cause. The cause is that words co-occur with similar ones. This means that word-break combinations are just another samples of local repetition.

As shown by the histogram above, this is not the case at least for the most prominent combination -y.q-. It is not caused by words co-occurring with similar ones. On the other hand, I guess your argument works for initial-initial repetition and final-final repetition.

(28-06-2019, 02:43 PM)Torsten Wrote: You are not allowed to view links. Register or Login to view.A general principle of the self-citation method is to copy glyph sequences and to avoid obvious repetitions by replacing glyphs with similar ones (see Timm 2015, p. 5). 

This idea seems compatible with pairs like t/k or r/s, but 'q' cannot systematically replace (or by replaced by) any other symbol: no other symbol behaves similarly. If you replace a random symbol with 'q' you are almost certain to obtain an illegal word (see You are not allowed to view links. Register or Login to view.). But if you consider the idea that the glyph is added as a prefix to a word staring with o-, when following -y, you could get something closer to what can be observed.

(28-06-2019, 05:06 PM)MarcoP Wrote: You are not allowed to view links. Register or Login to view.

(28-06-2019, 02:43 PM)Torsten Wrote: You are not allowed to view links. Register or Login to view.My point is that initial-initial repetition, final-final repetition, and word-break combinations have the same cause. The cause is that words co-occur with similar ones. This means that word-break combinations are just another samples of local repetition.

As shown by the histogram above, this is not the case at least for the most prominent combination -y.q-. It is not caused by words co-occurring with similar ones.

You misunderstood what I mean with "[font=Tahoma, Verdana, Arial, sans-serif]words co-occur with similar ones". Word generation rule 3 is "Combine two source words to create a new word" (see Timm & Schinner 2019, p. 10). Rule 3 means that [/font]it is possible to copy words or parts of words. See for instance the sequence <You are not allowed to view links. Register or Login to view.> in line 18v.P.2. I would argue that <okshy> is a copy of <ot> (modifed to <ok>) from the previous line and also a copy of <shy>. I would also argue that the next token <qokchy> is a copy of the final <-y> in <okshy> (modified to <q->) and also of the whole <okshy>-token (modified to <okchy>).

[font=Tahoma, Verdana, Arial, sans-serif][font=Tahoma, Verdana, Arial, sans-serif][font=Tahoma, Verdana, Arial, sans-serif]

(28-06-2019, 05:06 PM)MarcoP Wrote: You are not allowed to view links. Register or Login to view.[/font][/font][/font]On the other hand, I guess your argument works for initial-initial repetition and final-final repetition.
[font=Tahoma, Verdana, Arial, sans-serif][font=Tahoma, Verdana, Arial, sans-serif]

[/font][/font]

See for instance sequences like <You are not allowed to view links. Register or Login to view.> in line f99r.P2.7 or <qokedy.qokeedy.qokoy.qotedy.otedy.qokedy.qokedy.qokal> in line f108r.P.31.

[font=Tahoma, Verdana, Arial, sans-serif]

(28-06-2019, 05:06 PM)MarcoP Wrote: You are not allowed to view links. Register or Login to view.[/font]

(28-06-2019, 02:43 PM)Torsten Wrote: You are not allowed to view links. Register or Login to view.A general principle of the self-citation method is to copy glyph sequences and to avoid obvious repetitions by replacing glyphs with similar ones (see Timm 2015, p. 5). 

This idea seems compatible with pairs like t/k or r/s, but 'q' cannot systematically replace (or by replaced by) any other symbol: no other symbol behaves similarly. If you replace a random symbol with 'q' you are almost certain to obtain an illegal word (see You are not allowed to view links. Register or Login to view.). But if you consider the idea that the glyph is added as a prefix to a word staring with o-, when following -y, you could get something closer to what can be observed.

This is exactly my point. It is not possible to replace <q> with another glyph since <q> didn't work like other glyphs.