05-11-2025, 09:20 PM
(This post was last modified: 05-11-2025, 09:20 PM by Jorge_Stolfi.)
[quote="tavie" pid='60856' dateline='1723077705']
Thanks Tavie!
But that is a lot of statistics and questions to tackle all at once. Maybe we can consider just one topic -- like the distribution of Ch in Herbal-A -- and see whether we can understand that anomaly better?
For starters, we must take into account that the natural line-breaking algorithm causes the first word of each line to be longer than average, while the last 1-3 words tend to be shorter than average. The most common words of any language tend to be short, and the frequency of a character is dominated by its occurrence in the most common words. (For instance, "t" and "e" are very common in English partly because they occur in "the", "to", "it", "at", "be", "me", "he", etc.) Therefore, this side effect of natural line breaking can have an effect on the frequencies of line-initial letters.
So we should start by asking how much of the line-initial anomalies, specifically the frequency of Ch as line initial, could be explained this way.
The first step would be to compute the relative frequency of Ch as token-initial in Herbal-A, but separately for the tokens of each length L = 1, 2, 3, ... (Here and elsewhere the length is the ideal approximate geometric width of the written word, counting o as width 1; so that Ch and t have width 2, and aiin has width 3, etc -- even if these combinations are considered single letters for statistics purposes.) Obviously the frequency will be zero for L= 1, but also for L = 2, since Ch is rare as a word by itself. I wonder what the plot will look like for other values of L.
If the frequency of word-initial Ch is significantly different depending on L, then we can proceed with the next test: simulate the Scribe with different page widths, and check whether the frequency of line-initial Ch, still occurs in this "re-broken" text. Namely, take each parag of the section under study, discard its head line, and join the other lines into a single line. Then run the simple line-breaking algorithm on that string of words, with various values of the max line length parameter W.
Again, the simple line breaking algorithm is: keep adding new words to the current line; until the (geometric) width of the line, plus 1 (for the space), plus the width of the next word exceeds W. Then break the line before that word. (An experienced scribe would look ahead a few words, and adjust the spacing and width of letters so that the broken line ends precisely on the right rail. But this sophistication is not necessary for this test.)
With any W, the average length Li of the line-initial words should be greater than the global average word length L; while the average length of the line-final words Lf should be less than Li. And the latter effect may even be significant for the next-to-last word of each line.
I predict that the frequency of Ch as line-initial will be anomalous for any reasonable value of W. The question is whether this cause alone would account for the anomaly observed in the VMS.
All the best, --stolfi
Thanks Tavie!
But that is a lot of statistics and questions to tackle all at once. Maybe we can consider just one topic -- like the distribution of Ch in Herbal-A -- and see whether we can understand that anomaly better?
For starters, we must take into account that the natural line-breaking algorithm causes the first word of each line to be longer than average, while the last 1-3 words tend to be shorter than average. The most common words of any language tend to be short, and the frequency of a character is dominated by its occurrence in the most common words. (For instance, "t" and "e" are very common in English partly because they occur in "the", "to", "it", "at", "be", "me", "he", etc.) Therefore, this side effect of natural line breaking can have an effect on the frequencies of line-initial letters.
So we should start by asking how much of the line-initial anomalies, specifically the frequency of Ch as line initial, could be explained this way.
The first step would be to compute the relative frequency of Ch as token-initial in Herbal-A, but separately for the tokens of each length L = 1, 2, 3, ... (Here and elsewhere the length is the ideal approximate geometric width of the written word, counting o as width 1; so that Ch and t have width 2, and aiin has width 3, etc -- even if these combinations are considered single letters for statistics purposes.) Obviously the frequency will be zero for L= 1, but also for L = 2, since Ch is rare as a word by itself. I wonder what the plot will look like for other values of L.
If the frequency of word-initial Ch is significantly different depending on L, then we can proceed with the next test: simulate the Scribe with different page widths, and check whether the frequency of line-initial Ch, still occurs in this "re-broken" text. Namely, take each parag of the section under study, discard its head line, and join the other lines into a single line. Then run the simple line-breaking algorithm on that string of words, with various values of the max line length parameter W.
Again, the simple line breaking algorithm is: keep adding new words to the current line; until the (geometric) width of the line, plus 1 (for the space), plus the width of the next word exceeds W. Then break the line before that word. (An experienced scribe would look ahead a few words, and adjust the spacing and width of letters so that the broken line ends precisely on the right rail. But this sophistication is not necessary for this test.)
With any W, the average length Li of the line-initial words should be greater than the global average word length L; while the average length of the line-final words Lf should be less than Li. And the latter effect may even be significant for the next-to-last word of each line.
I predict that the frequency of Ch as line-initial will be anomalous for any reasonable value of W. The question is whether this cause alone would account for the anomaly observed in the VMS.
All the best, --stolfi