14-11-2025, 08:28 PM
More about the size bias of line-breaking:
The simplest line-breaking algorithm for running text is: keep writing words on the current line, until you get to a word that, if written, would run past the right rail. In that case, break line before that word, and continue as before on the next line.
It should be easy to see why that algorithm makes a line break much more likely before a long word than a short one.
To estimate the effect precisely one could use a word Markov or order 2 of 3, pipe the output though that line-breaking algorithm, and run the desired statistics. Or take the VMS text, join the lines of each parag into a single line, and pipe it through that line-breaking algorithm with a very different line width.
But that basic algorithm could be elaborated in a number of ways. First, the scribe choose to could split words in order to save vellum. In that case, it may leave some indication of the split (m?) at the end of the first line, or at the beginning the second one (y? q?) or both, or neither. Even if he leaves no mark, the statistics of the line-initial words will be different because they will include word suffixes in addition to whole words. And if he splits the word only between syllables, as we do today, both parts will be morphologically similar to whole words.
In another independent elaboration, when the scribe gets near the end of the line, he will look ahead 2-3 words, choose the line break, then stretch or shrink the writing so that the line will end on the right rail. (Modern word processors will stretch or shrink the spaces, but a Medieval scribe also stretch or shrink the characters themselves.) As a result, words near the end of the line are more likely to be improperly split or joined in the transcription files.
And a scribe could use abbreviations when necessary to squeeze one more word before the line break. The VMS Scribe probably could not read the text, but the Author may have told him that he could abbreviate aiin or aiiin as am if he needed to.
Also, as in many manuscripts of the time, the scribe may have placed some special mark (y? q?) at the start of a line whenever there was a sentence of sub-paragraph break anywhere within that line.
And here is another rather far-fetched idea. Suppose the language was tonal (not necessarily Asian; I gather that Swedish is tonal too, for instance). That is, the pitch pattern along a word would change its meaning. One way to record tones in such a language is to insert special symbols -- like digits, or a/o/y -- in the words to indicate the pitch level. Thus, for example, the Mandarin word "lǎo", with the "dipping" tone, could be written as "l2a1o3" (This system still used by linguists to discuss tone systems in a language-independent way.) Then, within the same line, one could save some ink by inserting those pitch codes only when there was a change of pitch. That is, instead of "b2a1o3 b3a4o4 b4a1o1" one could write just "b2a1o3 b3a4o ba1o". But after a line break the scribe may have felt necessary to insert a pitch code, even it there was no change from the previous line, for the benefit of the reader...
There may be many more processes like those above that result in anomalous statistics at the start and end of lines. And several of them may be at work in the VMS. Untangling them will require more than just staring at tables of statistics. One should try instead the "scientific"method: formulate an hypothesis about a possible cause of the anomalies, then devise the simplest statistical test that could prove or disprove that hypothesis...
All the best, --stolfi
The simplest line-breaking algorithm for running text is: keep writing words on the current line, until you get to a word that, if written, would run past the right rail. In that case, break line before that word, and continue as before on the next line.
It should be easy to see why that algorithm makes a line break much more likely before a long word than a short one.
To estimate the effect precisely one could use a word Markov or order 2 of 3, pipe the output though that line-breaking algorithm, and run the desired statistics. Or take the VMS text, join the lines of each parag into a single line, and pipe it through that line-breaking algorithm with a very different line width.
But that basic algorithm could be elaborated in a number of ways. First, the scribe choose to could split words in order to save vellum. In that case, it may leave some indication of the split (m?) at the end of the first line, or at the beginning the second one (y? q?) or both, or neither. Even if he leaves no mark, the statistics of the line-initial words will be different because they will include word suffixes in addition to whole words. And if he splits the word only between syllables, as we do today, both parts will be morphologically similar to whole words.
In another independent elaboration, when the scribe gets near the end of the line, he will look ahead 2-3 words, choose the line break, then stretch or shrink the writing so that the line will end on the right rail. (Modern word processors will stretch or shrink the spaces, but a Medieval scribe also stretch or shrink the characters themselves.) As a result, words near the end of the line are more likely to be improperly split or joined in the transcription files.
And a scribe could use abbreviations when necessary to squeeze one more word before the line break. The VMS Scribe probably could not read the text, but the Author may have told him that he could abbreviate aiin or aiiin as am if he needed to.
Also, as in many manuscripts of the time, the scribe may have placed some special mark (y? q?) at the start of a line whenever there was a sentence of sub-paragraph break anywhere within that line.
And here is another rather far-fetched idea. Suppose the language was tonal (not necessarily Asian; I gather that Swedish is tonal too, for instance). That is, the pitch pattern along a word would change its meaning. One way to record tones in such a language is to insert special symbols -- like digits, or a/o/y -- in the words to indicate the pitch level. Thus, for example, the Mandarin word "lǎo", with the "dipping" tone, could be written as "l2a1o3" (This system still used by linguists to discuss tone systems in a language-independent way.) Then, within the same line, one could save some ink by inserting those pitch codes only when there was a change of pitch. That is, instead of "b2a1o3 b3a4o4 b4a1o1" one could write just "b2a1o3 b3a4o ba1o". But after a line break the scribe may have felt necessary to insert a pitch code, even it there was no change from the previous line, for the benefit of the reader...
There may be many more processes like those above that result in anomalous statistics at the start and end of lines. And several of them may be at work in the VMS. Untangling them will require more than just staring at tables of statistics. One should try instead the "scientific"method: formulate an hypothesis about a possible cause of the anomalies, then devise the simplest statistical test that could prove or disprove that hypothesis...
All the best, --stolfi
