Proposed Manuscript Chronology

[npcompl33t]

I think this result was to be expected. Let's see if I'm getting this right:



The horseshoe curve is IMO the algorithm trying to keep the triangle inequality in the N-space (not of J-S Divergence but of the matrix norm in the MDS), telling you: "Hey, Pharma is different than Herbal A and Bio / Stars, but it's not exactly a smooth segway all N=230 pairwise Jensen-Shannon Divergence distances considered, you know?"

I'm not entirely getting the "polynomial correction" step. You can flatten any polynomic function into a line by compositing it with another function afterwards. Are you basically interpolating the 3rd PC dimension with a 4 degree polynomial and then mapping/flattening that to a line? Because in that case I'd argue that'd be skewing the data just to fit a line, not particularly insightful

Your results don't look wrong, btw -- on the contrary, they're are consistent with René's, MarcoP's, Torsten's and others'. I just think you're basically retreading the same path, except for the line fit step, which I'm not quite understanding what you did

I agree that it is not unusual to find clusters corresponding to questions. This actually asks a different section. Can the relationships between those clusters and individual bifolia be explained by a continuous progression.

Your summary is close, the distance matrix rows are not being used directly as features, it is instead operating directly on the underlying pairwise representations, but otherwise you have the right idea.  Distance metrics, in particular where features tend to arise and then disappear entirely, will cause this type of effect, and is typically referred to as distance saturation. 

The horseshoe also does not automatically arise from having clusters or satisfying a triangle inequality. 

The example is like this, if you were to distribute points along a spiral path, these points will have a sequential ordering. Yet measuring distance along any one axis does not actually recover the correct ordering of these points. Measuring the distance in between the individual points shows that more distance exists than any single axis can explain, and individually you cannot recover the ordering of a points separated on this spiral from pairwise distances between these points. SOme points that sit close to one another, may actually be on different loops of the spiral, considerably separated by the path distance. PCA is going to squash all these distances down to primary axis distances. It would show that two points on different spiral loops are actually closer than their true path distance.

The curve is attempting to find the continuous path through these points, that explains all of the intermediate distances. Reconstructing the full path through the spiral requires flattening it out, and that is what the curve is doing. It is not strictly a polynomial curve, its a parameterized curve that uses cubic splines with eleven knots and a smoothing penalty. The polynomial representation relates to identifying the specific shape of the w pattern in the 1st vs 3rd dimension plot, as opposed to the more common s shape. 

The final straightening step is indeed just a change of coordinates, it is not evidence itself. The evidence is whether the fitted path describes other observations. In this case, the dispersion of the individual pages of the same bifolium is reduced with curve. 

In the spiral example, lets say as you move along the spiral, you create 4 points that are slightly randomly offset from the spiral rather than a single point on the spiral. You would expect a line that minimizes the overall variation of these points to be a better approximation of the full spiral. This is why i reported the fact the curve reduces the bifolio level spread.

This does not necessarily prove that the curve or the MDS1 dimension reflect chronology. However, time is a known mechanism that can result in the horseshoe shape, as with the flinders petrie example. In that example, the raw measurements produce the same horseshoe, and it was proven that by 'flattening the spiral', you can recover the chronology of the pots. This does not prove the voynich horseshoe gradient represents a chronology. It is a theory to explain the gradient, based on the bifolio/production level dispersion of points, combined with the fact time can cause the horseshoe shape. 

Does that make sense?

---

Response to your other questions:

1. Yes, first order transition probabilities, conditioned on the preceding glyph. P(a|b), the probability of the next glyph being b, given the current glyph a. This is a first-order Markov representation of the text.
   
2. I don't average the transition matrixes. I first calculate the JS divergence between the full matrixes, then average the distances. Averaging the matrices would blend different observations together obscuring contrast. I did this because i found that Early A, using full lines and word transitions separately resulted in different features being used. After inspecting these features, i realized they were capturing meaningful information the other lacked. The low text per page in Herbal A meant it was critical to capture as much distinguishing information as possible, so I elected to combine them in the above way. Doing so made my Herbal A clusters more stable than using either in isolation. The later sections that have more text per page were largely unchanged across either representation.
   
3. No, gradient here does not mean a mathematical derivative. It means an ordered progression in composition. "environmental gradient" is used in ecological ordination, where this type of pattern often appears. for example, species composition changing along a temperature gradient. "latent transition-profile gradient" would probably be a more accurate term. The glyph-transition profiles change along an inferred direction or curved path. The reconstructed sequence assigns positions along that path using normalized arc length. Interpreting this progression as production chronology is a further hypothesis, nothing in the math itself directly establishes the underlying cause of the gradient is time, that is my hypothesis. 
   

---

EDIT 2: Realized i never addressed this, but the horseshoe shape appears when data has one dominant ordering, where items close together on it are similar, and similarity falls off with separation. Noticing it is a legitimate reason to go looking for that ordering / attempting to trace a path. That is the motivation and reasoning for drawing the curve. Its not just randomly tracing curves through the space with no justification. Based on the appearance of the Guttman Effect, there should be an underlying dominant order. What that ordering represents is up for debate, I am proposing time.