About the construction of lines in the MS

[Jorge_Stolfi]

And one important point: I removed spaces and line breaks completely and worked only at the character level.

Well, I don't know whether I coded it right, but here is what I got:

   

For this plot I used my transcription of the Starred Parags section.  I discarded parag head and tail lines, deleted all word spaces, and mapped glyphs to glyph classes 
  Q = {q}, 
  O = {a,o,y}, 
  B = {ch,che,sh,she,ee,eee}, 
  K = {k, t, p, f, ke, te} 
  N = {m,im,n,in,iin,iiin,ir,iir}  , 
  D = {d,l,r,s}
Platform gallows cth etc were mapped to BK.  This mapping should reduce the impact of sampling error.

Then, I discarded all lines with less than 22 letters. Then for each distance d in 0 to 10, and each line, I picked a par of characters W1,W1 separated by d chars.  In one run (red plot) the pair was near the center of the line.  In the other run (blue line) the pair straddled the line break: that is, char W1 was floor(d/2) chars before the end of the line and char W2 was ceil(d/2) into the next line (even if it it was a tail line).

In total, each run collected ~350 pairs.  From this data I computed the predictability P12 of W2 given W1. This quantity was defined as E12-E2, the entropy E2 (in bits) of the overall distribution of the W2 char, minus the entropy E12 of the distribution of W2 for each possible char W1, averaged over the W1 with their respective frequencies.  The plot shows that as a fraction P12/E2.

Thus, what the graph shows is that, in the middle of the line, knowing W1 reduces the uncertainty of the next four characters by about 33%, 22%, 15%, and 6%, respectively.  Whereas, across a line breaks, W1 gives no information about the next four characters.  It is as if the line break was a gap of at least 4 characters.

It is puzzling that the predictability does not drop to zero when d increases beyond 4, but hovers around 2-3%, both inside lines and across line break.  

It could mean that there are two kinds of parags, say some that are generally K-rich and some generally K-poor; so if you see a K, there is a slightly higher than average chance that any other char in the same parag is a K.  

Or it may be a bug in my code...

However, the predictability in the first case seems to be determined largely by a few character pairs.  Like, W1='q' is almost always followed by W2='o', and that contributes a lot to P12 when d=0.  If I do the same plot but deleting the class codes Q and O, the predictabilities in mid-line drop to nearly zero:

   

In retrospect, that is quite understandable, since 'q' is practically non-existent near line end, and 'o' is rather rare.  Thus it is not so much that prophecy does not work across line breaks, but rather that the most successful "prophets" stay away from those places.

I also tried with W1 and W2 defined as digrams rather than individual characters.  At that point the sampling errors may be again too big.  But the same qualitative behavior was observed.

More later.

All the best,--stolfi