9 hours ago
@Jorge_Stolfi I'm going to try to respond to all of your points and questions, but let me know if I miss anything. I appreciate your continued dialog about this.
First, a small correction, you refer to the above charts as showing 'dozens' of curves, they show 1,600 curves, so several orders of magnitude off.
This is exactly correct, I'm glad the plots helped here. Collapsing to two curves F1 and F2 may be a bit of an oversimplification but I think for the purposes of our discussion it is fine. Just to be clear though, the saved fits contain combinations of:
So really it can't be reduced to just two fixed global sequences.
Your first example is correct: when H-1a projects around t-0.05, H-1b projects around t-2.05, and for the F2 curves it is the reverse. Neither F1 nor F2 shuffles H-1a and H-1b together. So F1 or F2 curve places 1a and 1b as occupying the same position.
This is reasonable for locally straight projections away from endpoints. However, projections onto a curved or endpoint limited path can complicate the straight-line approximation. Many projections land at curve bends, where there is no single tangent. That limits how literally we can apply the straight-line approximation.
For individual pages, within a single fit, the typical central 80% of positions spans approximately 10.5% of the curve for 1a and 11.0% for 1b. For pooled bifolia, it is lower, between 6-8% roughly.
1.5a is not unique elongated, its typical within-fit width is about 7% for pages and 6% for bifolia.
Ultimately the above plots only capture the 1st 3 dimensions of a 256d space, so its natural they don't show all or even the majority of the distances. I'll share the MDS5 plots below, as they capture another dimension specifically useful for evaluating the end orderings.
I specifically tested this, removing the 3 largest word types on an already text-light page only removes a few % points of the divergence. A handful of repeated words does not explain most of its difference. The differences come from different word construction, specifically use of the bigram ol (apart from the single ol words, removing those does not change this), along with eo, initial ch, and o-t.
Topic can produce distinct clusters. What it cannot produce is a gradient. For example, there is no reason that arranging topics in the order i've listed would result in chol's counts gradually decreasing at each step. On a broad scale, the exact amount of 1a vocabulary retained from step to step declines gradually at each step. Topic cannot explain this. Style can, if it is a function of chronology or time.
These were all checked, and not every page was placed into a section. That is why there is an 'unclassified' label on the initial charts i shared. Each page was placed into a group only if it made sense, the relative certainty was established for every individual assignment. It was placed in 3b, because that is legitimately its best match, in the MDS plots i shared it genuinely looks like it is a part of that grouping.
Again, you are also only viewing the top 3 dimensions in a 256d space, you do not have the information to make that call from just the above plots alone. I've shared the full 256d MDS space if you want to take a deeper look.
Ignoring pages that seem to cause contradictions is exactly what can lead to warped perspectives about the MS as a whole, I do not think sweeping away the difficult pages under the rug will lead to a better understanding of the MS.
First, a small correction, you refer to the above charts as showing 'dozens' of curves, they show 1,600 curves, so several orders of magnitude off.
(Yesterday, 03:40 AM)Jorge_Stolfi wrote: You are not allowed to view links. Register or Login to view.But another important information I get from this plot is that the set of all the curves is partitioned into two compact and well-separated families, F1 and F2. Thus the theory of discontinuous projection that I had guessed to explain the split plot of B-4a and S-4c is unnecessary. The two parts of each plot are just the t values for the curves of F1 and for those of F2.
This is exactly correct, I'm glad the plots helped here. Collapsing to two curves F1 and F2 may be a bit of an oversimplification but I think for the purposes of our discussion it is fine. Just to be clear though, the saved fits contain combinations of:
- 1a before/after 1b
- bio 4a before/after stars 4c
- Herbal B 3a before/after Herbal B 3b
So really it can't be reduced to just two fixed global sequences.
(Yesterday, 03:40 AM)Jorge_Stolfi wrote: You are not allowed to view links. Register or Login to view.Do you mean that, for curves from F1, subsection H-1a projects around t = 0.05 and H-1b projects around t = 2.05; while for F2 curves, it is the reverse -- H-1a projects around t = 2.05 and H-1b projects around t = 0.05?
Or is the case that, for F1 curves, both subsections project near t =0.05, while for F2 curves both project around t = 0.25?
If it is the latter, that is evidence that, in that sub-pace at least, the two sections are practically coincident.
Your first example is correct: when H-1a projects around t-0.05, H-1b projects around t-2.05, and for the F2 curves it is the reverse. Neither F1 nor F2 shuffles H-1a and H-1b together. So F1 or F2 curve places 1a and 1b as occupying the same position.
(Yesterday, 03:40 AM)Jorge_Stolfi wrote: You are not allowed to view links. Register or Login to view.In that item I was referring to the pages of one cloud, not to two clouds.
Let's fix on sub-section H-1a,. For each curve C, all pages p of cloud 1a projected onto the curve at positions t(p,C) very close together around some mean t value tbar©; say 0.05±0.01. (Isn't that so?) (Except for an outlier about 1/3 the way down the list; let's ignore that). We can replace that curve C by a straight line segment which is tangent to the curve at that mean value tbar©,. The projections of each page p on that line segment will be practically the same,and thus all within 0.05±0.01.
The fact that all pages p have projections tbar(p,C) = ~0.05±0.01 says that the extent of the cloud H-1a along the direction of that axis is about ±0.01 times some length L that is approximately the length of the whole curve.
For the plot in question you used dozens of curves; that is, you projected each page on dozens of different axes. For every curve C' (IIUC) the projections t(p,C') of all pages p were close together, only around some different value tbar(C',1a) that varied between ~0.00 and ~0.20 depending on C'. Which means that, along all those directions in R^256, the extent of the cloud H-1a is equally small, namely about ±0.01 times L.
That is why I drew H-1a as a small smudge on my sketch. Maybe H-1a is much longer in other directions of R^256, but that elongation does not show up in your plots.
And the same applies to every other sub-section except 1.5a. The fact that, for each curve C, the pages of the sub-section S project to a narrow range of t values tbar(C,S)±0.01 says that the extent of that sub-section S, along any of the tangent directions of those curves, is ±0.01 times L.
Sub-section H-1.5a is a bit different, because for each curve C the projections t(p,C) of the points in that cloud are spread over a longer range, say tbar(C,1.5a)±0.03. That is why I drew that section as a smudge that is elongated in the direction of the curves.
This is reasonable for locally straight projections away from endpoints. However, projections onto a curved or endpoint limited path can complicate the straight-line approximation. Many projections land at curve bends, where there is no single tangent. That limits how literally we can apply the straight-line approximation.
For individual pages, within a single fit, the typical central 80% of positions spans approximately 10.5% of the curve for 1a and 11.0% for 1b. For pooled bifolia, it is lower, between 6-8% roughly.
1.5a is not unique elongated, its typical within-fit width is about 7% for pages and 6% for bifolia.
Ultimately the above plots only capture the 1st 3 dimensions of a 256d space, so its natural they don't show all or even the majority of the distances. I'll share the MDS5 plots below, as they capture another dimension specifically useful for evaluating the end orderings.
(Yesterday, 03:40 AM)Jorge_Stolfi wrote: You are not allowed to view links. Register or Login to view.Page You are not allowed to view links. Register or Login to view. explains that girls bathing on blue water pools should watch out for aardvarks. They come out of their burrows (where they hide from aasvogels) on warm afternoons to mate and gorge on aaliis berries. The text warns that they should not be confused with aardwolves. It explains that there are four types of aardvark, the blue aardvark, the teal aardvark, the red aardvark, and the pink aardvark (the colors being purely theoretical, not actual). Each type can be scared away by wearing a shower-cap of the proper color and reciting "by Aaron and Aalar, I command AAALL aardvarks to stay AAAWAY! Aaah! Aaah! Aaah!" The page is an outlier only because of all those "aa".
Being an outlier means that it has some rather unusual digraph frequencies. If you want to understand its puzzle, you should identify the anomalous digraphs, then look for the words that contain them. You may find that its anomaly is due to a single "unusual' word, or maybe a few such words, that are used quite a number of times on that page, and nowhere else. (I hereby propose the term "aardword" for that concept.)
I specifically tested this, removing the 3 largest word types on an already text-light page only removes a few % points of the divergence. A handful of repeated words does not explain most of its difference. The differences come from different word construction, specifically use of the bigram ol (apart from the single ol words, removing those does not change this), along with eo, initial ch, and o-t.
(Yesterday, 03:40 AM)Jorge_Stolfi wrote: You are not allowed to view links. Register or Login to view.You should consider at least possible (if not expected) that some (if not all) of the statistical differences between pages and sections are due to topic, style, etc -- not to time. Then You are not allowed to view links. Register or Login to view. and other "hard to place" pages will be no big mystery.
Topic can produce distinct clusters. What it cannot produce is a gradient. For example, there is no reason that arranging topics in the order i've listed would result in chol's counts gradually decreasing at each step. On a broad scale, the exact amount of 1a vocabulary retained from step to step declines gradually at each step. Topic cannot explain this. Style can, if it is a function of chronology or time.
(Yesterday, 03:40 AM)Jorge_Stolfi wrote: You are not allowed to view links. Register or Login to view.Seriously, that page obviously does not belong to Herbal, A or B. It should be put into Bio, even if it is an outlier there too.
By the way, you should just ignore the pages that cannot be clearly classified into any of the major sections sections by their image contents, including f1r, f57v, f58r, f58v, f76r, f85r1, f85r2, f86v6, f86v5, f86v4, f86v3, You are not allowed to view links. Register or Login to view. from line 31 on, and of course f116v. I guess that you want to include those pages because you see them as intermediate stages between the main sections. But you should check that afterwards, if and when you think you figured out the order of the main sections. Until then, including those pages in the analysis will only distract and confuse.
These were all checked, and not every page was placed into a section. That is why there is an 'unclassified' label on the initial charts i shared. Each page was placed into a group only if it made sense, the relative certainty was established for every individual assignment. It was placed in 3b, because that is legitimately its best match, in the MDS plots i shared it genuinely looks like it is a part of that grouping.
Again, you are also only viewing the top 3 dimensions in a 256d space, you do not have the information to make that call from just the above plots alone. I've shared the full 256d MDS space if you want to take a deeper look.
Ignoring pages that seem to cause contradictions is exactly what can lead to warped perspectives about the MS as a whole, I do not think sweeping away the difficult pages under the rug will lead to a better understanding of the MS.