Proposed Manuscript Chronology

[RadioFM]

I think this result was to be expected. Let's see if I'm getting this right:

You're calculating the transition probabilities on a page by page basis. Torsten and others have shown in the past that neighbouring pages read similarly (albeit with some exceptions, likely due to misbinding).

The Jensen-Shannon Divergence as I understand it is symmetrical, so when taking pairwise distances between the probability distributions from each page, you'll have a diagonal full of zeros (each distribution has a distance 0 to itself) and smaller numbers the more similar the distribution is. Therefore, the vector of Jensen-Shannon Divergence of any Herbal A folio will have small numbers (small distances -- similar glyph groupings -- similar glyph transitions) on the coordinates respective to other Herbal A folios, a little bit larger values on Cosmo coordinates (Cosmo reads somewhat similar to Herbal A), then Pharma, and so on in increasing magnitude.

For example you would have the first 3 rows of the pairwise Jensen-Shannon Divergence matrix looking like this (allow me to re-order and group folia):

   Herbal A            ···   Herbal B  ···                     Pharma  ···            Cosmo   ···             Bio ···                   Stars ··· 
[   0    0.05   0.04   ··· 0.29  0.55  0.66  0.77  ···  0.25  0.29  0.23 ··· 0.11  0.18  0.15  ··· 0.80  0.82  0.86 ··· 0.70  0.76 ··· ]
[ 0.05   0      0.02   ··· 0.27  0.53  0.68  0.71  ···  0.29  0.31  0.28 ··· 0.07  0.10  0.08  ··· 0.83  0.83  0.84 ··· 0.74  0.71 ··· ]
[ 0.04  0.2      0      ··· 0.31  0.47  0.63  0.69  ···  0.24  0.26  0.27 ··· 0.09  0.13  0.07  ··· 0.79  0.81  0.85 ··· 0.68  0.77 ··· ]

As you can imagine, in an N~230 (1 per folio?) dimensional space, each folio will have lower values on those coordinates mapping to its own section. Naturally the points cluster for each section with some in-betweens, probably consistent with RZ's extended languages, Torsten's self-citation theory and so on. So it's not surprising to find clusters of sections in some projection, in fact we expect PCA or MDS to do just that as best it can.

The horseshoe curve is IMO the algorithm trying to keep the triangle inequality in the N-space (not of J-S Divergence but of the matrix norm in the MDS), telling you: "Hey, Pharma is different than Herbal A and Bio / Stars, but it's not exactly a smooth segway/intermediate all N=230 dimensions considered, you know?"

I'm not entirely getting the "polynomial correction" step. You can flatten any polynomic function into a line by compositing it with another function afterwards. Are you basically interpolating the 3rd PC dimension with a 4 degree polynomial and then mapping/flattening that to a line? Because in that case I'd argue that'd be skewing the data just to fit a line, not particularly insightful

Your results don't look wrong, btw -- on the contrary, they are consistent with René's, MarcoP's, Torsten's and others'. I just think you're basically retreading the same path, except for the line fit step, which I'm not quite understanding what you did

---

PS I got some questions to better understand your methodology:

1. Glyph transition: Are you using a Markov Model (conditioned on the last glyph only?)
   
2. Glyph transition matrix: How do you merge/combine the folio and/or line level transitions into a single transition matrix per folio?
   
3. By "[environmental] gradient" you mean the gradient (1st order derivatives) on the R^N space of the Jensen-Shannon distances, correct?
   

Thanks