About the construction of lines in the MS

[Jorge_Stolfi]

How could the tendency of the second words of lines to begin disproportionately with [Sh] and [s] be explained as a byproduct of an algorithm for splitting lines based on word length?

Texts in natural languages normally have significant correlations between successive words.  That is,, the probability Pr(t[i] = Y | t[i-1] = X) is usually very different from Pr(t[i] = Y).

Thus, any process that affects the word frequencies at the start of the line will also affect the frequencies at the 2nd position.  And also at the 3rd position, 4th, and so on, but with decreasing magnitude.

In the case you mention,it must be that words starting with sh are more likely to occur after a longer word than a shorter one.  

I did a simulation of this effect by creating two token streams, A and B.  Bot have 50% of "short" tokens with 2 letters and 50% of "long" tokens with 7 letters.  In file A, each token is chosen independently, so that there is no correlation between successive tokens.  In file B, a 1st-order Markov process was used to get a long token be followed by another long token 70% of the time (instead of 50%).  As a consequence, a short token was also followed by another short one 70% of the time.   Here is a sample of the TLA-justified text:

ytchedy dy ar dy ol qokaiin ol ol ol ol ytchedy shochdy
qokaiin shochdy ytchedy dy ar dy qokaiin qokaiin ol dy ar ol
ar ol dy ar ar ol ol ytchedy qokaiin shochdy ytchedy dy
shochdy shochdy shochdy ytchedy ytchedy qokaiin qokaiin dy
ytchedy shochdy ytchedy qokaiin shochdy qokaiin ytchedy ar
shochdy dy ol dy ol ol ol dy ol ar dy qokaiin qokaiin
shochdy dy dy ol ytchedy ytchedy shochdy ol ol dy shochdy dy
ar ol ol shochdy shochdy shochdy qokaiin ar shochdy qokaiin
dy dy dy dy ytchedy shochdy dy dy shochdy ytchedy qokaiin
ytchedy shochdy ytchedy shochdy qokaiin ytchedy ytchedy
shochdy ol ol shochdy qokaiin dy dy ytchedy ytchedy ar ol

Here are the statistics of word length depending on position along the line:

   

The plots were truncated to positions 1-7 only.  Since all lines of the text have at least 7 tokens, this avoids the selection bias artifacts showed in the previous posts.

The left plot shows that TLA visibly increases the length of words in positions 1 to 4 on the lines when the word lengths are positively correlated.  Whereas, when there is no correlation, the effect is limited to the first word of the line.

The right plot surprised me again, because it shows that positive length correlation can increase the average length of the last word, above the general average word length.  I am struggling to understand the math cause of this result, but it seems real.

All the best, --stolfi