Proposed Manuscript Chronology

[npcompl33t]

I fully agree with you stolfi on your first 2 plots. neither of those plots are guttman horseshoes, and neither would justify treating it as a gradient.

Your point on the S plot however, is not accurate. 

A Guttman Horseshoe can manifest a more complex shape than an S. All that is necessary is that it is a harmonic oscillation. It must exhibit symmetry passing through the axis, which an S does. 

However, more complex shapes do this as well. A w, is a harmonic of the S. Rather than pass through the axis 3 times, it passes through 4 times. 

Voynich manifests the W shape, as shown below, and in my OP.


   

[Mauro]

I'm impressed by your work @npcompl33t. It's also fascinating to come to know of what look to be relatively recent development in dimensionality reduction (I was stuck to PCA...). I have bookmarked the Diaconis at al. arXiv article you referenced and I'll try to read it with calm, but the introduction rather agrees with what you affirm: dominant gradients create the horseshoe shape.

If I understood well, you also found support for @LisaFaginDavis theory that the VMS 'unit' is the bifolio, am I right?

And a (possibly silly, sorry) question: could the gradient also be in reverse, first Bio and last Herbal A?

[Jorge_Stolfi]

Your point on the S plot however, is not accurate. A Guttman Horseshoe can manifest a more complex shape than an S. Voynich manifests the W shape, as shown below, and in my OP.

The point of my third figure is not the shape of the smudge, but its continuity and smoothness.  

In order to claim that the points are best modeled as a "smoke trail" -- a compact prob distribution D(t) that depends on a gradually varying parameter t -- rather than a "cloud" or multiple "clouds", the density of points must change gradually along the curve C(t) of the centroid of that distribution.  Like in my figure #3.  Preferably in a bell-shaped fashion, with the density increasing and decreasing smoothly along the curve, with a single density peak around the middle of the curve. 

But that is not the case for the VMS page points.  The distribution is much more like my figure #2, with 4-5 dense clumps -- coinciding with the major sections -- that are separated by areas of much reduced density.   The clouds are even elongated in different directions, that are not parallel to the alleged direction of the "gradient".

The few missing folios cannot explain those gaps between the clouds.

The distributions overlap somewhat (at least in that 2D projection) which may give the impression of a singe cloud -- if you do not actually count dots per area.  But that is not enough to justify a "smoke trail" model instead of a "five clouds" model. 

Moreover, we do not expect any text statistics to show a smooth consistent change from page to page, whether the pages are considered in chronological writing order, in intended reading order, or in the order which they were eventually bound.  Between sections, the statistical differences are expected to be dominated by the  change of topic and the changes of style that go with that.  Within most sections, we expect random variations around a single mean.  

In the Herbal sections, for example, the text on each page surely is about the plant depicted there.  Quite probably the plants are in random order, like in the "You are not allowed to view links. Register or Login to view." In that case, the word frequencies of two consecutive pages should be just as similar as those of two distant pages -- and all statistics I have seen used for page comparison and sorting are determined by the word frequencies.

A similar argument applies to the Astro/Cosmo, Zodiac, Pharma, and Starred Parags sectons.

The Bio/Bathing section is the only one where we could expect some statistical continuity if the pages are considered in the intended reading order. The reasoning is that, if the text starts digressing on one page about how scalp massage benefits an ingrown toenail, it is somewhat likely that "scalp", "massage", and "toenail" will be mentioned also on the next page.  However, this effect is likley to be small compared to the sampling noise and to the noise due to sub-topics that are exhausted within a single page.   Thus, if one tries ordering the pages of Bio by any statistical similarity criterion, the resut may look like a continuous gradient, but the order and the gradient will be nonsensical -- like what one gets by ordering books by size.

All the best, --stolfi

[npcompl33t]

I'm impressed by your work @npcompl33t. It's also fascinating to come to know of what look to be relatively recent development in dimensionality reduction (I was stuck to PCA...). I have bookmarked the Diaconis at al. arXiv article you referenced and I'll try to read it with calm, but the introduction rather agrees with what you affirm: dominant gradients create the horseshoe shape.

If I understood well, you also found support for @LisaFaginDavis theory that the VMS 'unit' is the bifolio, am I right?

And a (possibly silly, sorry) question: could the gradient also be in reverse, first Bio and last Herbal A?

Thank you! Yes, this shows that all pages on a bifolio are a functional unit, LFD's proposal is in my opinion absolutely confirmed. Honestly, it is so apparent in the data I assumed her findings had become widely accepted and didn't need any qualifications.

Your 2nd question is not silly, and yes, absolutely the gradient could be in reverse, A path can be found but the order can't be specified from the data, I am making the assumption that Herbal A is the start and Bio is the end.

[npcompl33t]

The point of my third figure is not the shape of the smudge, but its continuity and smoothness.  

That's fair, but ultimately the smoothness is a function of the granularity of data which doesn't exist with blocks of text like this. It is going to look chunky.

In order to claim that the points are best modeled as a "smoke trail" -- a compact prob distribution D(t) that depends on a gradually varying parameter t -- rather than a "cloud" or multiple "clouds", the density of points must change gradually along the curve C(t) of the centroid of that distribution.  Like in my figure #3.  Preferably in a bell-shaped fashion, with the density increasing and decreasing smoothly along the curve, with a single density peak around the middle of the curve. 

They do, both in the 2 dimensional mds1 vs mds2 and mds1 vs mds3 charts, but also in the full 256 dimensional space, which is what the curve shows. 

Moreover, we do not expect any text statistics to show a smooth consistent change from page to page, whether the pages are considered in chronological writing order, in intended reading order, or in the order which they were eventually bound.  Between sections, the statistical differences are expected to be dominated by the  change of topic and the changes of style that go with that.  Within most sections, we expect random variations around a single mean.  

You are absolutely correct, and that is why this deserves close attention, because it implies something about the txt we have been missing.

If you look at my post in the last page, i showed an example of this occurring in Napier's logbooks, for the exact reasons you would expect. He gradually changes the way he reports certain things over the course of multiple years.

If you You are not allowed to view links. Register or Login to view. , which i hope you will take a look at,  i show this gradual process exactly, with words specific word examples.

If you split the text into the sections i've identified, and treat them as a sequence, you will find that between any 2 nodes, there is a high degree of overlap, but that overlap decreases according to the number of steps, creating a gradient of word turnover from the start of the manuscript to the end.

This is exactly the Currier A->Currier B translation we have been discussing and that your previous work had suggested exists, it just happens over a series of steps, a gradient of cumulative changes.

The next step is to start to model these changes, similar to how linguists model phoneme changes. Once we establish the correspondence between all words, i think that things are going to develop quickly. I'm not saying that a solution will be found but I think this has been a significant confound in previous work. self citation vs evolution of the underlying script should present differently and we may be able to distinguish those.

[Jorge_Stolfi]

 For a horesshoe to mechanically appear, a single linear ordering, must be the most significant part of the feature space. Otherwise it would not  appear. By appearing, it means there is a meaningful order.

There seems to be a lot of superstition and quackery in the field of data analysis.  

For example, there is a big sub-area (called Technical Analysis or TA) devoted to analysis of stock prices, where the main goal is to predict the price P[t+1] of a stock at time t+1 (next month, day, millisecond) from its past history P[t], P[t-1], ....  TA practitioners have defined dozens of patterns that are supposedly important for that goal, such as "trends", "flags", "support", "MACD", "head and shoulders", "EMA crossings", "Fibonacci retracings", "Anchored VWap", etc.  And there are thousands of thick books about TA, both general and devoted to pushing a particular tool.  

But 99.99% of that theory is nonsense -- produced by the same motives that drive quacks in any field, and sustained by pareidolia, wishful thinking, and that old human desire:

• Tiger got to hunt, bird got to fly;
  
• Man got to sit and wonder 'why, why, why?'
  
• Tiger got to sleep, bird got to land;
  
• Man got to tell himself he understand. (K. Vonnegut)
  

The 0.01% of TA that is not nonsense is the single result that is supported by proper analysis, the Geometric Brownian model.  Which says that log(P[t+1]) = log(P[t]) + R[t], where each R[t] is a random variable with approximately Gaussian distribution, independent of all previous prices P[t-1],P[t-2], ...  That is, stock prices are effectively unpredictable, and their past histories do not give any information about their future, except that the price tomorrow will be somewhat similar to the price today.

Well, that "Guttman Horseshoe" sounds like "superstition" too.   Consider this fact: Let c[1], c[2],... c[m] be any m data points of R^d, and p[1], p[2], ... p[m] be any m points of the plane R^2, in any order.  If d >= m, there is an affine  projection of R^d to R^2 that projects each c[k] to p[k].

(An affine projection is a linear projection plus a translation.  The claim is false in some degenerate cases, when the data points happen to lie all in a small subspace of R^d; but this should not be the case for the VMS page data.)

In particular, for any 5 clouds of points in R^d with d >= 5, and any permutation of the indices 1..5, there is an affine projection of R^d to R^2 that will map the centroids of those clouds to five points on the parabola y = x^2, in that specified order and equally spaced in x.  In other words, we can always get a "Guttman Horsehoe"  with the clouds in any arbitrary order.

Here is an example with d = m = 4.  The data points are 4 clouds whose centroids are the vertices of a regular tetrahedron in R^4.  Within each cloud the points are drawn from an almost round 4D Gaussian distributon. So there is no intrinsic order in the data; all 4! = 6 orders of the 4 clouds are equivalent.  

Here is a projection of those data points where the four centroids project to the corners of a square:
   

But here is another projection where the same clouds lie on a "horseshoe" curve, apparently in order red-green-cyan-purple:
   

All the best, --stolfi

[npcompl33t]

 For a horesshoe to mechanically appear, a single linear ordering, must be the most significant part of the feature space. Otherwise it would not  appear. By appearing, it means there is a meaningful order.

There seems to be a lot of superstition and quackery in the field of data analysis.  

I linked to several papers and even a multivariate analysis textbook on this, i stand by that statement and i think the sources I linked to support it. If you disagree with them, I'm not sure we are going to agree on this. I hope you will read my last message and the new post on the evolution sequence. I think there is quite a substantial bit of evidence now this gradient exists, and I show in the other post how it manifests at the word level. I think there will be a lot of interesting findings that result from this, and I hope to have your help discovering and validating them.