The evolution of P/F vs T/K raises important questions

[Jorge_Stolfi]

Yes

Thanks again for the nice plots!

About the second set of plots (puff counts as function of head line length in glyphs), you say

Herbal A does not have the highest first-line rate. Herbal B–3b has 7.34 occurrences per 100 glyphs, compared with 4.46 for 1a, 5.76 for 1b, and 6.14 for transitional 1.5a. The two Stars groups differ. Stars–4c is low at 3.40, while Stars–3c is 4.47, almost identical to Bio–4a’s 4.49. These include benched forms; plain-only comparisons are also provided.

But those ratios are not true correlations.  They are artifacts from using a strictly linear model Y = AX in the fitting, rather than an affine model Y = AX+B.  Even if X and Y are completely independent (uncorrelated) random variables with means Xavg and Yavg, fitting a strictly linear model will yield an apparently strong correlation with ratio A = Yavg/Xavg.

Looking at the dot clouds of the second set of plots, and trying to do a "visual" affine fit Y = AX+B, I would say that there exists a definite correlation in Herbal-A, Herbal-B,and Pharma, but only a weak correlation in Bio, and no significant correlation in Stars.

In fact, the weak apparent correlation in Bio seems to be due to a few outliers: five parag head lines with fewer than 25 glyphs, and one head line with 9 puffs.  Excluding those outliers still may give an affine correlation Y = AX+B with positive A, maybe even larger than that of the strictly linear model; but the correlation index should be very low.

Thus, before drawing any conclusions from those plots, we should check whether those outliers are legit or are data errors.

That brings up one important point that affects any analysis that depends on parag size: several parag breaks in the transcription file are wrong, either because of transcription uncertainty or because the Scribe himself goofed.   Should we count the two lines at the top right of You are not allowed to view links. Register or Login to view. as one parag? Are the first 11 words of the last parag of You are not allowed to view links. Register or Login to view. part of its head line, or are they the names of the 11 nymphs in the lower left pool, which the Scribe confused with the text of the parag?

This caveat applies also to the Stars section.  With an affine model Y = AX+B, if we exclude the single outlier of Gof 4c (part of Stars), with ~55 glyphs and 9 puffs, there seems to be no correlation in the remaining dots.   It seems that, in that section, a parag head line has about three puffs on average, independently of its length.

Ditto for Golf 3c (also Stars), If we exclude the three head lines with fewer than 40 characters.  Each of those "parags" must consist of a single very short line.  Are they perhaps the three "titles" at the bottom of f104v, f105r, and f108v? 

By the way, several parag breaks in the Stars section, in particular, are rather doubtful.  In other sections, when the digital transcribers see a set of consecutive full-width lines followed by a short line, they will mark that as a single parag.  But in the Stars section sometimes what seems to be a single parag by that criterion will have two or three stars in the margin.  That is usually taken as a sign that the "parag" is in  fact  two parags, such that the tail line of the first parag happened to be full-width.  (Or that the Scribe decided to  copy both parags without a line break, for some reason.)   Then transcribers may insert a parag break mark using secondary clues, like the interline spacing, the presence of puffs or taller-than-normal gallows.  But we can bet that the guess is often wrong; in which case the head line of the second parag is not in fact a head line.

Similar caveats apply to Golf 2a "Late A/Pharma" (?).   There is an apparently distinct cluster of head lines with less than 20 glyphs.  Which "parags" are they from? Could they be in fact labels, incorrectly marked as parag text in the transcription file? Anyway, if we exclude those very short "head lines", the rest does not seem to show any correlation.  That is, it looks like the head line in that section has about 2 puffs on average, independently of its length -- even in those parags that extend across two pages.

With respect to the first set of plots (puff counts vs parag size in words) you say

Section differences don’t explain away the paragraph-size association.

What I see in these plots is:

• In Golf 1a, 1b,and 1.5a (mostly Herbal-A?) there is no significant relation between total puff count and total word count in the parag.  Each parag has ~3 puffs, independently of its size.  This could be explained by the use of puffs being mostly confined to the parag head lines.
  
• In Golf 2a "Late A/Pharma" (?) there may or may not be a relation.  But it is pointless to analyze that "section", because it is an unjustified mix of crab apples and osage oranges -- pages that obviously belong to two different "books", with different topics, organizations, styles, and quite probably different creation epochs.
  
• In Golf 3a and 3b (Herbal-B, presumably), there does seem to be a significant correlation.  Maybe even more with an affine model Y = AX+B fit than with the linear Y = AX one.  This suggests significant use of puffs for purposes other than highlighting the head line or the first word of the parag.   But this correlation may be an artifact of incorrect parag breaks (see below).
  
• In Golf 4a (Bio), the apparent correlation may be dominated by the few outliers -- extra-long parags, which again may be the result of missed parag breaks in the transcription file. 
  
• In Golf 3c and 4c (Stars), each of those outliers is surely one of the "parags" with two or more stars in the margin, which surely must be two or more parags that were mashed together in the transcription file.  The longest single-star parag is f105v.32 with only 74 words,even counting commas as word spaces.  The second-longest single-star parag may be f104v.1 with 55 words.   The average size of single-star parags is 30-40 words.  In fact there are three large blocks of full-width lines on pages f108v, You are not allowed to view links. Register or Login to view. and You are not allowed to view links. Register or Login to view. that have respectively 11, 11, and 10 stars in the margin, and thus are probably the junction of that many original parags that were mashed together by the Scribe for some reason. Those must be the three extreme outliers in that plot of Golf 4c.  When two or more parags are joined, both their total puff counts and their total word counts get added together -- resulting in a large but spurious contribution to the apparent correlation of the two quantities.  If we exclude the obvious outliers -- parags with more than ~75 words -- there seems to be no correlation: each parag seems to have ~2.5 puffs, independently of its size.
  

By the way, the single outlier in the first plot of Golf 4c (a head line with 55 glyphs and 9 puffs) is line f111r.36, the first line after one of those three big mashup blocks. Maybe that is when the Scribe was scolded for ignoring parag breaks, and compensated by exaggerating the head-puffing of the next parag?

This underscores the importance of carefully checking the data before applying any data analysis method to it, and examining it again before drawing conclusions from its results.  The old adage "garbage in, garbage out" was coined in the 1950s precisely for this situation -- data analysis by computers.

All the best, --stolfi