Chronological Path Stability/Uncertainty Analysis

[Jorge_Stolfi]

@Jorge_Stolfi I'm going to try to respond to all of your points and questions, but let me know if I miss anything.

Thanks for the extensive reply and the new graphs!

But another important information I get from this plot is that the set of all the curves is partitioned into two compact and well-separated families, F1 and F2.

Just to be clear though, the saved fits contain combinations of:

• 1a before/after 1b
  
• bio 4a before/after stars 4c
  
• Herbal B 3a before/after Herbal B 3b
  

So really it can't be reduced to just two fixed global sequences.

So you mean that there are three or more families of curves, such that the curves of different families make different combinations of choices at each of the "river bifurcations" visible on those curve plots?

The existence of multiple families of fitted curves, that yield different t orders for the pages and sections, is an argument against your thesis -- that the chronological order of writing (COW) can be deduced from the oredering of the points in R^256 space.  Right?  

Let v be the dominating direction of the current fitted curves.   In those plots, the projection of v run mostly left to right.  What would happento the fitted curves if you stretched the data along a direction u in R^256 that is perpendicular to v?  Wouldn't the new fitted curves run mostly in the direction u? Wouldn't that completely scramble the t order of the sections?

That brings up some moe fundamental questions I have about the general methodology and the Jensen-ShennongShannon distance.  Will ask them later.

when H-1a projects around t-0.05, H-1b projects around t-2.05, and for the F2 curves it is the reverse.

Ah, OK, so H-1a and H-1b do seem to be well-separated clouds in R^256 space.

Let's fix on sub-section H-1a,.  For each curve C, all pages p of cloud 1a projected onto the curve at positions t(p,C) very close together around some mean t value tbar©; say 0.05±0.01. (Isn't that so?) ...The fact that all pages p have projections tbar(p,C) = ~0.05±0.01 says that the extent of the cloud H-1a along the direction of that axis is about ±0.01 times some length L that is approximately the length of the whole curve.

For individual pages, within a single fit, the typical central 80% of positions spans approximately 10.5% of the curve for 1a and 11.0% for 1b. For pooled bifolia, it is lower, between 6-8% roughly.

Sorry, I am confused.  Do you mean that, for a single curve, the projections of the H-1a pages span a range of ~0.10 in t values? Not ~0.02, as I had assumed above?

In You are not allowed to view links. Register or Login to view. there are some white vertical streaks across all color swabs.  I had assumed that they were due to the use of a small number of curves.  That assumption influenced my misunderstanding that the t projections of the H-1a pages, for each single curve, spanned only ±0.01.  But you say you had thousands of curves, and now I see that those white bands are equally spaced in t.  So now I guess that they are plotting artifacts, perhaps due to something as banal as writing the data out to some intermediate file with too few decimal digits.  Is that so?

1.5a is not unique elongated, its typical within-fit width is about 7% for pages and 6% for bifolia.

All other color swabs in that plot are roughly rectangular.  With the new information, I interpret this fact a meaning that the t-ordering of the pages of those sections is drastically different for different curves, even within the same family.   Which would mean that the order of the projections of those pages along a direction u in R^256 space changes drastically even with small changes in u.  Which seems to imply that the pages of each section are not mostly aligned along some direction v, but are scattered randomly in a roundish cloud.  No?

But H-15a seems to be different: its swab is a parallelogram instead of a rectangle.  So I suppose that not only its pages project in the same order along any curve, but in fact that order is highly correlated with the order of those pages in your data file.  Which order is this?

Page You are not allowed to view links. Register or Login to view. explains that girls bathing on blue water pools should watch out for aardvarks.  ... The page is an outlier only because of all those "aa". ... If you want to understand its puzzle, you should identify the anomalous digraphs, then look for the words that contain them.

I specifically tested this, removing the 3 largest word types on an already text-light page only removes a few % points of the divergence. A handful of repeated words does not explain most of its difference. The differences come from different word construction, specifically use of the bigram ol (apart from the single ol words, removing those does not change this), along with eo, initial ch, and o-t.

Is that page still an outlier if you delete all spaces (points, dashes, commas)?  The Scribe seems to have been inconsistent with spaces around some elements, such as ol, al, or, ar.

Seriously, that page obviously does not belong to Herbal, A or B.  It should be put into Bio, even if it is an outlier there too.

[f84v] was placed in H-3b, because that is legitimately its best match, in the MDS plots i shared it genuinely looks like it is a part of that grouping.

It may be the best match, but it is still a very poor match.

But if the anomalies of You are not allowed to view links. Register or Login to view. are so scattered, then I retract my idea of an "aardword" effect and the proposal to move it to Bio.  Instead, I propose that You are not allowed to view links. Register or Login to view. is not really part of Bio, although it may be related to it (like an epilogue about the history of the section). It is a section by itself, another "unknown" section, like the next page f85r1.  The illustrations logically belonged to the previous page, but did not fit there so the Scribe spilled them over to the unrelated page You are not allowed to view links. Register or Login to view. (something that happens all the time on books and papers even today).

It may also be that only the lower half of You are not allowed to view links. Register or Login to view. is anomalous and is not part of Bio proper (like the lower half of You are not allowed to view links. Register or Login to view. does not seem to be part of the Starred Parags section proper).  That half seems to be a single long parag, twice as long as the average parags of other Bio pages.

You should consider at least possible (if not expected) that some (if not all) of the statistical differences between pages and sections are due to topic, style, etc -- not to time. Then You are not allowed to view links. Register or Login to view. and other "hard to place" pages will be no big mystery.

Topic can produce distinct clusters. What it cannot produce is a gradient. For example, there is no reason that arranging topics in the order i've listed would result in chol's counts gradually decreasing at each step.

If you listed the pages in order of mean t-value along some curve, and you fitted that curve to a bunch of statistics, and those statistics happen to be correlated with the frequency of chol, then the frequency of chol will tend to vary gradually along that ordering.  

It is again like sorting books by height, and observing that the number of words per page (along with many other book properties) varies gradually along the resulting ordering.

On a broad scale, the exact amount of 1a vocabulary retained from step to step declines gradually at each step. Topic cannot explain this. Style can, if it is a function of chronology or time.

But, again, what page ordering are you assuming when you say "step to step"?  The order of mean t-position along the fitted curves?

You should just ignore the pages that cannot be clearly classified into any of the major sections sections by their image contents, including f1r, f57v, f58r, f58v, f76r, f85r1, f85r2, f86v6, f86v5, f86v4, f86v3, You are not allowed to view links. Register or Login to view. from line 31 on, and of course f116v. ...  including those pages in the analysis will only distract and confuse.

Ignoring pages that seem to cause contradictions is exactly what can lead to warped perspectives about the MS as a whole, I do not think sweeping away the difficult pages under the rug will lead to a better understanding of the MS.

Well, I think that in fact it is generally better to ignore outliers until one understand the inliers.

• If an outlier was generated by the same process that generated the inliers (like sampling from a distribution with a fat tail), ignoring the outlier will not make much difference to understanding the process;
  
• But if an outlier was generated by a glitch or external interference in the process (like the Author accidentally including a love letter in his draft), including it in the analysis will only distract, distort the fitted curves and PCA directions, create bogus correlations, etc.
  

All the best, --stolfi