Proposed Manuscript Chronology

[MarcoP]

could the gradient also be in reverse, first Bio and last Herbal A?

yes, absolutely the gradient could be in reverse, A path can be found but the order can't be specified from the data, I am making the assumption that Herbal A is the start and Bio is the end.

I agree with making that assumption. Though the binding order is unreliable, I think it still provides meaningful information:

• The only page with colored initials was bound as the first page (and it's Currier A).
  
• The only page with a mostly blank verso was bound as the last page (and the recto is Currier B).
  
• The binding order is partly consistent with Currier A/B: strongly A at the start, dubious pages in the middle (Cosmo/Zodiac), strongly B in the second half (Bio, Stars). The attached graph plots page numbers vs MDS1 as provided by npcompl33t (You are not allowed to view links. Register or Login to view.): correlation -0.66. I hope I haven’t messed up things too much.
  

In my opinion, this evidence points to “A before B” rather than the reverse.

   

[npcompl33t]

 I am quite open to the idea that there may be a mostly "orthographic" mapping of words to words that makes the word frequencies of those two sets mostly equal (apart from sampling noise).  Have you seen You are not allowed to view links. Register or Login to view.?

Yes, i have seen that. You have shared your post with me multiple times now, and I have even incorporated the same chart of yours directly into my app. Your work has directly influenced my analysis and my conclusions here.

I analyzed the Black Book of Basingwerk's famous orthographic shift earlier in this thread, and included the same MDS plot. It does not show a horseshoe like voynich. However, that is because it has a 'sudden' orthographic shift. 

In the Napier example, the way he reports his case logs slightly varies over the course of the first few years of his logs. This also creates a horseshoe, for an orthographic like reason, changing conventions. It also shows not *all* words need to be impacted for there to be a gradient.

LFD has speculated that the VMS was never intended to be bound, and were instead loose collections of bifolia. My work here shows there is a temporal sequence, but the scale of time is unknown. What if, for example, these were all written over the course of months or even a few year? Maybe the scribes were recording some previous collection that was written over a much longer time span. Maybe this was some sort of medieval equivalent to a weekend side project that extended for some time. In that case, orthographic shifts would appear gradually.

I think far to much effort has been spent arguing over whether the gradient exists rather than trying to understand what could actually explain such a gradient. There are reasons why an invented script would show this kind of drift aside from self citation. Real languages drift like this over certain time resolutions. 

If it truly is self citation as opposed to some sort of drift over true semantic meaning, then the changes that occur should appear randomly and should not be concerned with preserving information across the gradient. 

And be careful to not impose, explicitly or unconsciously, the assumption that the A/B transition is gradual. 

I did not encode this assumption. My analysis started exactly as you described, comparing Herbal A/B. My initial assumption was the opposite of this, and you can You are not allowed to view links. Register or Login to view. over the course of the analysis in the thread i created.

The conclusions of this thread directly came from the analysis you are describing, which did not encode that assumption.

Remember also that character-based statistics (character and digram frequencies, entropies, etc.) are mere projections of word statistics 

I am very aware of this. My analysis used a transition matrix / first order markov representation of the text, but it is still word projections. I've shown however, as with both the Napier and BBoB example that this does capture real meaning in the way you would expect. 

Besides, the VMS is unlikely to be all original work by the Author.  While the language seem to be mostly the same throughout the book (apart from A/B and smaller differences), it is very likely that each section is a translation, transcription, or adaptation of a different "ur-book", written by a different author in a different place and epoch.

This is exactly the thing that would cause a gradient to appear. If the manuscript was indeed written over a wider period of time, if its conventions were not fixed, or it was actively being developed while written, all of these could introduce the type of continuous gradient the horseshoe pattern implies. 

I strongly believe that VMS research as a whole would benefit from shifting the debate away from 'is there a gradient' to 'what mechanisms could cause this gradient and how can we distinguish between them'. 

[magnesium]

could the gradient also be in reverse, first Bio and last Herbal A?

yes, absolutely the gradient could be in reverse, A path can be found but the order can't be specified from the data, I am making the assumption that Herbal A is the start and Bio is the end.

I agree with making that assumption. Though the binding order is unreliable, I think it still provides meaningful information:

• The only page with colored initials was bound as the first page (and it's Currier A).
  
• The only page with a mostly blank verso was bound as the last page (and the recto is Currier B).
  
• The binding order is partly consistent with Currier A/B: strongly A at the start, dubious pages in the middle (Cosmo/Zodiac), strongly B in the second half (Bio, Stars). The attached graph plots page numbers vs MDS1 as provided by npcompl33t (You are not allowed to view links. Register or Login to view.): correlation -0.66. I hope I haven’t messed up things too much.
  

In my opinion, this evidence points to “A before B” rather than the reverse.

We also must remember that the handwriting is large in A and small in B. Assuming for the sake of argument that a single generative process created the VMS, IMO it'd be likelier for the writing to shrink in size as the scribe gets more comfortable writing in Voynichese, rather than the reverse.

[Rafal]

IMO it'd be likelier for the writing to shrink in size as the scribe gets more comfortable writing in Voynichese

But don't forget there was probably more than one scribe involved.

In such case the scribe didn't change his style. There were several guys, each with his own style that they didn't change.

All this thread seems to talk about some "evolution" while there could be no evolution, just each guy to his own, doing it from start to end his personal way.

[Jorge_Stolfi]

The authors of that paper DID recover a gradient from that horseshoe that matches up almost perfectly with established indexes of liberalism vs conservatism. How do you explain that if it contains no sequential ordering?

How are those "indexes of liberalism vs conservatism" computed? Perhaps from ... the way those representatves vote?

Sure, for any data set and any axis, the data points "contain" a sequential ordering along that axis.  See again the book example: if you measure the height and width, the books will have a sequential ordering along the height axis, and also a sequential ordering along the width axis.  And one may observe that the two variables are strongly correlated, and that would be a significant observation.  But the ordering along either axis will say nothing except the banal observation that any set of numbers "contains" a sequential ordering.

Have you noticed the following comment in a previous post of mine?

Let c[1], c[2],... c[m] be any m data points of R^d, and p[1], p[2], ... p[m] be any m points of the plane R^2, in any order.  If d >= m, there is an affine  projection of R^d to R^2 that projects each c[k] to p[k].

Can you see what this fact implies about the "Guttman Horseshoe" and the page analysis?

In particular, if you have 240 pages of any book, and for each page you compute any 256 independent statistics, that fact implies that there is a projection of that data onto the plane that will produce the following plot:
   
In fact, for any of the 240! possible assignments of the 240 pages to the 240 dots in that plot, there is a projection that takes each page 256-vector to its assigned dot.

Another way of looking at this fact is that doing "agnostic" multivariate data analysis is likely to produce meaningless results if the number of data coordinates is greater than (or even close to) the number of data points.  Like fitting a polynomial of degree 10 on 12 noisy data points.   

Moreover, the idea of extracting some "intrinsic sequential ordering" for a set of data points is equivalent to identifying some "principal" axis of variation.  It has the problem that the result cannot be invariant with respect to linear transformations of the data vectors.  A simple change of scale along some axis can radically change the direction of such a "principal axis" (like that given by PCA) -- and consequently the "intrinsic sequential ordering" of those points.

Consider again that four-cloud example that I just posted.  Since the centroids are the vertices of a regular tetrahedron in R^4, any ordering of the four clouds is equivalent to any other.  But  for any of the 4! = 6 possible orders of the centroids there is a  linear transformation of R^4 -- a modest stretching in the proper direction -- that causes the "principal axis" of the data set to be such that, along that axis, the clouds are clearly sorted along that axis
   

Thus the concept of "principal axis" is useful only when the Euclidean metric is meaningful for the data space.  Like, when the data points are points of physical 3-space, where the Euclidean distance has a special meaning.  Then only linear transformations of the data that preserve that metric allowed; and then one can define "princpal direction" in a way that commutes with those transformations.  

In contrast, when the Euclidean metric is not meaningful for the data space, the concept of "principal direction" is meaningless too.  That is the case, in particular, when the coordinates of the data vectors are measured in different arbitrary units, like (length,mass,temperature),.  A mere change of unit on any axis can completely change the direction of the "principal direction" determined by PCA, SVD, or any other method.

And this is almost certainly the case for those 256  page statistics that you said you are using.  If they are frequencies of specific words, characters, or digraphs, each statistic has a different amount of noise, hence a change in its value is not equivalent to the same change in the value of some other statistic.

A more meaningful concept, because it is invariant under linear transformations of the data, is the concept of "linear separator" of two sets X, Y of points of R^d.  It is an oriented hyperplane in R^d that maximizes some measure of how accurately it separates the two sets, like the number of X points on the positive side plus the number of Y points on the negative side.  This measure can be taken as the "linear separability" of the two sets.  It is invariant under any linear transformations of the data space, not just isometries.

And, for page statistics typically used in the VMS page order analysis, the clouds of any two major sections have high linear separability.  Even sections that appear consecutive in that "Guttman Horseshoe" plots.  There is a hyperplane in the data space that separates the two clouds with only a small number of pages on the "wrong" side.

Moreover, if one projects the data points of those two sections on the axis orthogonal to that hyperplane, and builds histograms of those projections, one will get two well separated Gaussian-like humps, with a low valley between them.  Meaning that the two sections are not a slice of a single continuous "smoke trail", but two independent clouds, that may overlap to some extent.

Another important point is that most popular methods of determining "principal direction", like PCA or SVG, will not reliably identify the direction S of linear separation (the axis othogonal to the linear separator).  That is because those method are designed under the assumption that the Euclidean metric of the data space is meaningful.  The linear axis S also not the direction of the line between the two centroids.  To find S one must use algorithm that will specifically look for it.

Here are two clouds of points on the plane R^2 that are almost completely separated by an hyperplane (the green line), with a clear band of low density along that separator. 
   
Note that the principal direction determined by PCA (the largest eigenvector of the covariance matrix) and the centroid-to-centroid direction CCD are useless, because they are almost perpendicular to the direction SEP along which the clouds are seen to be separated.

All the best, --stolfi

[magnesium]

IMO it'd be likelier for the writing to shrink in size as the scribe gets more comfortable writing in Voynichese

But don't forget there was probably more than one scribe involved.

In such case the scribe didn't change his style. There were several guys, each with his own style that they didn't change.

All this thread seems to talk about some "evolution" while there could be no evolution, just each guy to his own, doing it from start to end his personal way.

Absolutely, hence the "assuming a single generative process" qualifier on my statement. It could simply be that Davis's Scribe 1 just writes bigger than the other scribes.

[npcompl33t]

How are those "indexes of liberalism vs conservatism" computed? Perhaps from ... the way those representatves vote?
Sure, for any data set and any axis, the data points "contain" a sequential ordering along that axis.

Either there is a dominant axis or there is not. Anyone familiar with modern politics knows that it is bipolar and regardless of how people vote, most could identify based on policy positions where a representative would fall on the spectrum. Every data set does not contain a dominant sequential ordering, that is core to this entire debate and the literature I shared on Horseshoes in particular. 

I'm not claiming all PCA always produces meaningful gradients, or that PCA dimensions correspond to real world traits as opposed to abstract mathematical relationships. I agree with you on all these points.

Where we disagree appears to be your assertion that a horseshoe does not contain a dominant ordering that directly ties to a single linear feature like time or temperature.  I shared multiple studies and even textbooks where people identify discuss this at length, and recover and identify the dominant sequential ordering from horseshoes in a number of real-world contexts. I feel extremely confident my findings here are legitimate, and supported by the broader research.

[MarcoP]

IMO it'd be likelier for the writing to shrink in size as the scribe gets more comfortable writing in Voynichese

But don't forget there was probably more than one scribe involved.

In such case the scribe didn't change his style. There were several guys, each with his own style that they didn't change.

All this thread seems to talk about some "evolution" while there could be no evolution, just each guy to his own, doing it from start to end his personal way.

We know that the reason for "dialect" differences isn't different subjects (as defined by different illustrations) nor different scribes. Scribe 1 wrote both Herbal A and Pharma. But Pharmacese is quite different from the language of most of Herbal A (in particular, bigram EVA:eo is much more frequent). Also, a few HA pages that were bound together with Pharma have a language that is more similar to Pharmacese than to the rest of Herbal A (e.g. bifolio 87-90).

So there is no single Scribe1 "personal way": his style deeply changes between the pages bound at the start and those bound near the end of the manuscript.
 
See plots for CUVA SO/EO (EVA:cho/eo) at the bottom of You are not allowed to view links. Register or Login to view. (dots colored by image type -red HA, yellow Pharma- and labelled by scribe/folio-number)-

[Rafal]

We know that the reason for "dialect" differences isn't different subjects (as defined by different illustrations) nor different scribes.

Do we?

From VoynichArchive search engine:

Tell me how Currier A and Currier B are related to supposed different scribes

Prescott Currier identified two distinct statistical patterns in the Voynich Manuscript's text, which he termed Language A and Language B, and also discerned two primary scribal hands, Scribe 1 and Scribe 2, in the botanical section of the manuscript. He noted a direct correlation between Language A and Scribe 1, and Language B and Scribe 2. [url=https://voynicharchive.com/#citation-t0-2][/url]Currier's work suggested that these differences in text statistics corresponded to variations in handwriting, implying the involvement of at least two authors.For example, Currier A rarely features the bigram /ed/, while Currier B uses it frequently. While Currier initially focused on two hands, he believed there could be as many as eight scribes involved. More recent analysis by Lisa Fagin Davis has identified five distinct scribal hands, with Scribe 1 primarily writing in Language A, and Scribes 2, 3, 4, and 5 writing in Language B

I realise it is more subtle but if I understand it correctly it is the main, strongest effect and the rest are nuances and refinements.

[Jorge_Stolfi]

I think far to much effort has been spent arguing over whether the gradient exists rather than trying to understand what could actually explain such a gradient.  ... I strongly believe that VMS research as a whole would benefit from shifting the debate away from 'is there a gradient' to 'what mechanisms could cause this gradient and how can we distinguish between them'.

And I strongly believe exactly the opposite: far too much effort has been wasted trying extract a gradual change (between sections, bifolios, of pages) from page statistics, when the data itself says that there is no such gradual change.

What your plots show is that the pages of each section form a compact cloud that could be fairly well modeled by a single n-dimensional Gaussian distribution.  Apart from sampling noise, the point density has a single maximum, and drops off smoothly from that point in any direction.  

That cloud may be somewhat stretched along some direction, either in space on on a 2D projection.  The points can be ordered by their position along that direction of maximum extent (or any other direction selected by some "sacred" data analysis method).  But that ordering is not significant, because the direction of maximum extent of the cloud (and therefore that ordering) can be changed to any other arbitrary direction, by changing the projection and/or by applying a linear transformation of the data space that squeezes or stretches the cloud appropriately.  That is possible because there is no "natural" distance metric for that data space.

Therefore, from the per-page statistics data we cannot deduce any intrinsic order, temporal or otherwise, for the pages of each section.  

That said, if we now look at the clouds of any two distinct sections X,Y (perhaps with a few exceptions), it can be seen from the plots that the clouds are well separated: each cloud has a distinct point of maximum density,and there is a region of significantly lower density between them.   

The separation is almost certainly more evident in the data space, and could be made more evident with a proper projection (see the last image in my previous plot). 

Those two sets of page data vectors cannot be properly modeled as samples from a single Gaussian distribution, nor as two slices of a single elongated "smoke trail".  

The only minimally acceptable model is the sum of two Gaussian distributions, such that the separation between the means is large enough to create a low "saddle" between them. 

Therefore, what we can say so far is that all the pages of a section X are statistically similar to each other, modulo random variations, with no obvious ordering or evolution; and they are noticeably different from those of most any other section Y.  

A priori, it seems likely (but by no means certain) that there is a temporal order between sections.  That is, the Author probably composed and clean-copied the whole draft of a section X, which he gave to the Scribe, before starting to do the same for another section Y.   

Can we assume that the statistical differences between X and Y are due to the mere passage of time, that somehow caused the Author to change his spelling rules and/or word usage frequencies?  

Well, no.  Most pairs of sections X,Y have very different text structures (paragraphs vs. text rings vs recipes vs. continuous text etc.) and almost certainly very different topics.  These differences, by themselves, are more than enough to explain the statistical differences.  Recall, again, that the frequencies of characters and digraphs in a txt are determined by their presence or absence in the most common words in that text.  Any changes that are the result of mere passage of time (like the Author becoming gradually more fond of daiin and less inclined to use -edy words) would be easily swamped by changes due to those other factors.

It is even possible (not likely, but definitely possible) that the Author created two sections X,Y in parallell - writing some pages of X, then some pages of Y, then back to some pages of X, etc.  But, as he switched back and forth, so would his word usage frequencies, because they would be determined by the topics; and so would the per-page statistics.

But if the clouds of sections X, Y, and Z lie more or less in a straight line, in that order, can't we infer that Y was probably created between X and Z?  Or lies somehow between the two in some variable other than time (like the use of astrology in the discussion, or the education level expected from the intended readers)?

We may tentatively make that hypothesis if the three clouds are mostly aligned in the data space R^n.  In that case, they would be aligned in any projection.  But if they are not aligned in R^n, then, by choosing different projections, we can make them look aligned on the plot, and in any of the 3! = 6 possible orders -- with any of them apparently between the other two.  

And I bet that, in fact, You are not allowed to view links. Register or Login to view..

[A compilation of multiple ur-books] is exactly the thing that would cause a gradient to appear.

As explained above, that is in fact a thing that would cause the clouds to lie in random, non-aligned position in data space -- and mask the effects of mere passage of time.

All the best, --stolfi