14-11-2025, 03:21 PM
Let me also remind people of the recent finding that the line-breaking algorithm used by scribes (not just on the VMS, but in any language and any epoch, even today) has the side effect of making the first word of each line longer than average, and the last few words shorter than average.
This phenomenon alone can have a significant effect on word frequencies at the start of the line, because the most frequent words in a text tend to be short. For instance, if a running English text is broken into lines in the simplest possible way, we can expect that the words "the", "and", "is", "it" will occur more rarely at the start of lines than in the text as a whole. Conversely they will occur more frequently at the end of lines.
That is why it is important to run all statistical analyses on a control (non-VMS) sample whenever possible. The anomalies at line-start may be real, but may be due to this and other causes -- not to any semantic role of line breaks.
For example, here is a quick test using the Portuguese novel that I sent to @Quimqu a while ago:
These frequencies were obtained by feeding the running text, where each paragraph was formatted as a single line, through a trivial line breaking program (Linux "fmt --split-lines") that broke each parag into lines of 72 chars max. Then taking each line with 10 words of more (there were ~4800 of them) and separating the first word from the other words, and computing the frequencies of both.
Note that the the frequencies of short words like "a", "o" are systematically higher in the second column than in the first. And I did not bother to exclude the parag head lines; this may explain why "que" (= "what", "which", "whose", "who", "that"...) is still about equally common in both sets, since it occurs often at the start of interrogative sentences, and hence at the start of parags.
This bias towards longer words can also affect the statistics of line-initial characters, since character frequencies are determined largely by their occurrence in high-frequency words.
All the best, --stolfi
This phenomenon alone can have a significant effect on word frequencies at the start of the line, because the most frequent words in a text tend to be short. For instance, if a running English text is broken into lines in the simplest possible way, we can expect that the words "the", "and", "is", "it" will occur more rarely at the start of lines than in the text as a whole. Conversely they will occur more frequently at the end of lines.
That is why it is important to run all statistical analyses on a control (non-VMS) sample whenever possible. The anomalies at line-start may be real, but may be due to this and other causes -- not to any semantic role of line breaks.
For example, here is a quick test using the Portuguese novel that I sent to @Quimqu a while ago:
Code:
# first nonfirst
#
0.04249 que 0.04354 que
0.04155 e 0.04039 a
0.03732 a 0.03443 e
0.03404 não 0.03131 de
0.02582 de 0.02779 o
0.02207 o 0.02185 não
0.01338 mas 0.01686 me
0.01150 para 0.01343 se
0.01150 se 0.01231 um
0.01103 era 0.01093 os
0.01103 é 0.01053 é
0.01033 capitu 0.01010 da
0.00986 como 0.00981 do
0.00962 um 0.00892 era
0.00915 os 0.00877 mas
0.00845 da 0.00850 as
0.00822 com 0.00848 para
0.00751 em 0.00826 com
0.00681 do 0.00826 eu
0.00681 por 0.00782 lhe
0.00634 as 0.00737 em
0.00610 ao 0.00654 uma
0.00610 eu 0.00565 ao
0.00563 me 0.00555 por
0.00563 uma 0.00552 minha
0.00516 na 0.00538 na
0.00493 quando 0.00534 no
0.00469 no 0.00526 mais
0.00446 mais 0.00519 como
0.00446 ou 0.00445 ou
0.00423 nem 0.00441 à
0.00399 dos 0.00420 capitu
0.00399 foi 0.00379 mãe
0.00399 à 0.00377 nem
0.00376 já 0.00356 ele
0.00376 minha 0.00352 foi
0.00376 também 0.00298 tudo
0.00352 agora 0.00288 só
0.00352 lhe 0.00281 dias
0.00352 só 0.00281 quando
0.00352 tudo 0.00279 casa
0.00329 pois 0.00279 ela
0.00329 sem 0.00277 dos
0.00329 tinha 0.00275 disse
0.00282 ainda 0.00275 meu
0.00282 antes 0.00275 olhos
0.00282 assim 0.00271 *
0.00282 ele 0.00261 já
0.00282 há 0.00261 ser
0.00282 josé 0.00259 mim These frequencies were obtained by feeding the running text, where each paragraph was formatted as a single line, through a trivial line breaking program (Linux "fmt --split-lines") that broke each parag into lines of 72 chars max. Then taking each line with 10 words of more (there were ~4800 of them) and separating the first word from the other words, and computing the frequencies of both.
Note that the the frequencies of short words like "a", "o" are systematically higher in the second column than in the first. And I did not bother to exclude the parag head lines; this may explain why "que" (= "what", "which", "whose", "who", "that"...) is still about equally common in both sets, since it occurs often at the start of interrogative sentences, and hence at the start of parags.
This bias towards longer words can also affect the statistics of line-initial characters, since character frequencies are determined largely by their occurrence in high-frequency words.
All the best, --stolfi
