About the construction of lines in the MS

[Jorge_Stolfi]

according to the data and as a summary to my coding in this thread, the lines seem to have some strong initial tokens, some strong final tokens and a middle region. But what I find more interesting is that each line seems independent from the next or previous one. The results for that are strong, indicating that there is almost no connection from the last words to the first words of the next line. This, under my opinion, doesn't happen in a "normal" text and is a clue that we should take into account.

Are you sure this result is real, and not just an artifact of sampling error?

I just tried looking at whether word k+2 can be predicted from word k, in the middle of the line and across line breaks.

Specifically, I took all 483 parag body lines (neither head nor tail) with at least 10 words in the Starred Parags section, considering commas as word spaces. Did two passes, (A) and (B)  From each line, I extracted a pair of words {W1,W2} where 
(A) W1 is word N/2-1 and W2 is word N/2+1 of the N words on the line; or
(B) W1 is word N-1 of the line and W2 is word 1 of the next line. 

I skipped one word because the last word of the line is known to be anomalous, due to abbreviations, compression, etc.  This might reduce the predictability a lot, but should have the same effect for (A) and (B).  

Anyway, this potential problem turned out to not matter.  What happened is that practically every pair {W1,W2} occurred only once in (A) and (B) together.  Thus, 483 pairs seem to be way too small a sample to determine whether W2 can be predicted from W1.

So, how can you say that definitely there is no correlation between words across a word break?  Should I try again without skipping one word between W1 and W2?

All the best, --stolfi