Can LAAFU effects be modeled?

[Jorge_Stolfi]

[Reposted here from the general Chat thread]

1: the frequency counts of characters at beginnings and endings of lines are markedly different from elsewhere. There are some characters that may not occur initially in a line. There are others whose occurrence is about one hundredth of the expected

There are three "non-functional" processes that can contribute to start-of-line and end-of-line anomalies:

1. When a scribe is transcribing running text, he tends to break lines more often before a long word than before a short one.  Thus the word frequencies at start-of-line, end-of-line, and middle-of-line will naturally be different.  And if word frequencies are different, character frequencies will usually be different too; since character frequencies are dominated by their occurrence in the most common words, and these tend to be very short.

2. When a scribe is writing text by hand, word and character spaces tend to be more stretched out near start-of-line and more cramped near end-of-line.   Thus modern transcribers tend to split words near the start-of-line and join them near end-of-line. 

This is what I find in Rene's file when I look for matches across line breaks of a word with a prefix of the next word, or with a suffix of the previous word:

== hea ==
1083 pairs checked, 11 repeats found
      2                daiin : daiin             
      1                    d : dchy                
      1                    s : sody               
      1                    y : ychain             
      1                  chy : chy           
      1                cthol : ol                 
      1              ydaiin : daiin             
      1              shdaiin : daiin             
      1            qokokchy : okchy             
      1            ycholdchy : y                 
== heb ==
338 pairs checked, 1 repeats found
      1                kary : y                 
== bio ==
734 pairs checked, 12 repeats found
      1                    l : lor               
      1                  ls : lshey             
      1                  ol : l                 
      1                  dar : dar               
      1                  oky : y                 
      1                  oly : y                 
      1                  qol : qol               
      1                chedy : y                 
      1                oroly : y                 
      1              oraisy : y                 
      1              qokaiin : qokaiin           
      1              qokeedy : qokeedy           
== str ==
1131 pairs checked, 9 repeats found
      2                  oky : y                 
      1                    y : ytaiin             
      1                  ol : olain             
      1                  ary : y                 
      1                otol : ol                 
      1                qoky : y                 
      1                ykal : ykaly             
      1          qopchedyldy : y


3. In You are not allowed to view links. Register or Login to view., there are no physical parag breaks as we use them today.  Instead, the start of a new "logical paragraph"  is marked by a red underline (and, bizarrely, by a red vertical stroke through the first letter).  But in some  cases the scribe also added a red pilcrow in the left margin, next to the line where the parag starts.

So perhaps something like that is happening in the VMS too.  Say, the draft had important parag or sentence breaks marked in some way.  The Scribe copied each mark wherever it fell on the line,  but also added some auxiliary mark at the start of the line, like an y or q. The ink used in those marks may have had some organic coloring added, like beet juice, that soon degraded without a trace, leaving the mark with the same brown ink as the text...

Maybe processes 1 and 2 are not sufficient to explain all the line-start and line-end anomalies, and maybe process 3 did not happen. But the point is that there may well be explanations for those anomalies that do not imply LAAFU.  That is, which left the Scribe still free to choose the line breaks without regard for the breaks in the Author's draft or for the meaning of the text. 

And there is evidence that the line breaks were indeed chosen by the Scribe as he was transcribing the text, without regard for its meaning or the breaks in the draft.

All the best, --stolfi