Opinions on: line as a functional unit

[Jorge_Stolfi]

I thought this was a theory proposed by Elmar Vogt and Ger Hungerink in 2012...but I didn't think it was proven as an explanation.

The effect itself is real. Someone posted histograms of first/mid/last word lengths to this forum not long ago.   And you saw my quick check on that Portuguese novel above.

But I had the impression only printed works were counted (is that wrong?), and it would be more interesting to see assessments of manuscripts, especially where longer words could in theory squeeze in at the end of the line by being abbreviated.

I don't recall whether t has been verified on the VMS, but I don't see why it would not hold for manuscripts too.   On manuscripts the scribe can squeeze the font itself to delay line breaking.  But even so a break is more likely to be unavoidable before a long word than a short one.

It would also be interesting to separate out paragraph start words (we see some really long ones, as well as short ones like "pol") from the line start pack since they are almost certainly not being wrapped round.

Indeed.  My advice is to exclude parag head lines and not look at them at all.  Whatever happens there is likely to be different and more complicated than what happens on the other lines.  So let's first understand the latter.  Then we can go back and study the head lines, using whatever we found about the body lines as a starting point... 

Another point I'd make is that while the word wrap concept of longer words not fitting at line end is a potential explanation for the word length discrepancies, I don't see it as being a comprehensive potential explanation for the glyph discrepancies.  It might have an impact, but I don't see the evidence for it being the primary explanatory factor for why word types at line start are different.

If the tokens at line start are longer than average, longer word types must have higher frequencies in that position than elsewhere, and the opposite must be true for shorter word types.  For instance, the frequency of qokeedy should be higher at line start than at mid-line, while the opposite should be true for ar.  But indeed it would be important to verify and quantify these differences for the VMS.  

We would expect the word types or glyph clusters "missing" at line end to be excessively popular at line start.  In many cases, we do not see this.  Something else is going on.

Indeed, it is not certain that these differences in word type frequencies will cause differences in character frequencies, but is certainly possible.   And even if the line breaking length bias turns out to be insufficient to explain the line-start anomalies, we would have to subtract its effects in order to understand the real anomalies and infer their causes.

One of the things I learned in the years that I spent watching cryptocurrencies is the term FOMO = "fear of missing out".  It is the tendency of many investors to put money into new things, even if they don't understand them, for fear of missing out on the next Apple or Facebook.  Maybe something like that is happening here too?  People being reluctant to focus on just one section, to discard head lines, and to collapse similar glyphs into classes, for fear of missing out on some important discovery? [Unicode emojis deleted by the forum's text editor]

All the best, --stolfi