- Options
- Login
- Desktop Style
Options
Opinions on: line as a functional unit
- Opinions on: line as a functional unit
-
RE: Opinions on: line as a functional unit
bi3mw > 11-12-2025, 09:32 PM
Thanks Jorge, the English test revealed a minor logical error. Very common, short whole words were counted as chunks (“the”). I have corrected the script accordingly. Here is the output:
www_line_start_end_pairs.xlsx (Size: 54.39 KB / Downloads: 1)
The table for the VMS with the corrections noted by @nablator. The combination qo → y is now far ahead.
RF1a-n-x7_line_start_end_pairs.xlsx (Size: 26.82 KB / Downloads: 2)
-
RE: Opinions on: line as a functional unit
Jorge_Stolfi > 11-12-2025, 11:07 PM
(11-12-2025, 08:50 PM)nablator Wrote: You are not allowed to view links. Register or Login to view.In manuscripts and printed books, line breaks don't cut syllables, usually.
Yes. But, AFAIK, there is no evidence that the VMS scribe would break lines in the middle of words, so that rule probably does not apply here.
Quote:There may be other rules in some cases.
There are rules for breaking before or after "and" and articles, I forgot which. There is also a rule against breaking between a number and a unit of measure.
All the best, --stolfi -
RE: Opinions on: line as a functional unit
Jorge_Stolfi > 11-12-2025, 11:11 PM
-
RE: Opinions on: line as a functional unit
bi3mw > 12-12-2025, 12:02 AM
(11-12-2025, 11:11 PM)Jorge_Stolfi Wrote: You are not allowed to view links. Register or Login to view.Thanks! So, what is the conclusion?
Unfortunately, I'm not there yet
What one could try is to write all the chunks line by line into a file and then calculate the expected start and end chunks. Then one could compare these with the actual output. I'll think about it.
-
RE: Opinions on: line as a functional unit
Bluetoes101 > 12-12-2025, 12:49 AM
I was thinking about this pattern while reading a book lately. It struck me that this sort of "green eggs and ham" talking is a trope of "learned (sometimes self proposed) men" towards others.
"All you that faine philosophers would be,
And night and day in Geber's kitchen broyle,
Wasting the chipps of ancient Hermes' Tree,
Weening to turn them to a precious oyle,
The more you worke the more you loose and spoile;
To you, I say, how learned soever you be,
Go burne your Bookes and come and learne of me."
It's not a song or poem, just Edward Kelly trying to blow his own trumpet. There's pages more, if you dislike your braincells I can share. -
RE: Opinions on: line as a functional unit
Jorge_Stolfi > 12-12-2025, 01:26 AM
(11-12-2025, 07:56 PM)tavie Wrote: You are not allowed to view links. Register or Login to view.
(11-12-2025, 07:41 PM)Jorge_Stolfi Wrote: You are not allowed to view links. Register or Login to view. The question that remains is how much of the "LAAFU" phenomenon on the VMS can be explained by this effect.
If indeed any (for the reasons I've already set out earlier in this post or another thread)
The mere line breaking process definitely created differences in the line-initial, line-medial, and line-final word distributions. It does that with any language, as long as the words have different lengths.
If the three word distributions are different, the statistics of glyphs and digraphs at the three locations are almost certainly different too.
I can believe that this Line Breaking Bias (LBB) was not the only cause of the differences we see in the VMS. I expect that there will be other causes too; like the Scribe using am as an abbreviation of aiin, or y as a calligraphic variant of o, or squeezing words spaces and stretching glyph spaces so that transcribers incorrectly join or split words. And maybe the line is indeed a functional unit.
One could test the effect of each of these possible causes, alone and then in combination. Then, if the LAAFU theory is true, one would hopefully get a better understanding of it after subtracting all those other factors.
To test and measure the LBB effect, for instance, one could- Take all parags in some hopefully homogeneous section, such as Herbal-A or Bio;
- Remove the parag head lines, and, for good measure, maybe also the second lines;
- Join all remainig lines of all parags into a single string of words;
- Insert line breaks with the trivial line breaking algorithm, for an arbitrary line width W;
- Compute the "LAAFU statistics" on the resulting text;
- Compare that with the "LAAFU statistics" of the same lines with original breaks.
All the best, --stolfi - Take all parags in some hopefully homogeneous section, such as Herbal-A or Bio;
-
RE: Opinions on: line as a functional unit
bi3mw > 12-12-2025, 06:06 AM
I thought about how to predict the start and end chunks. The result is as follows (if there is a flaw in my reasoning, please let me know):
A script reads a file in which each word is divided into start, middle, and end chunks. These parts are separated from each other. The middle chunks of each line are then converted into binary vectors to calculate the similarity between these lines. For each line, only those lines that achieve a certain minimum similarity (threshold value 0.6) are taken into account. This forms a group of similar lines. From this group, the most probable start and end chunks are predicted using weighted voting. Result:
Median group size: 25.0
Average similarity in groups: 0.636
Min similarity in groups: 0.600 (Threshold)
Max similarity in groups: 0.741
Exact matches (Actual == Predicted): 81 / 3885
Here is the output file: You are not allowed to view links. Register or Login to view.
The corpus used: You are not allowed to view links. Register or Login to view.
-
RE: Opinions on: line as a functional unit
bi3mw > 12-12-2025, 10:50 PM
Here is the cross-check with a shuffled corpus (words and lines):
Median group size: 15.0
Average similarity in groups: 0.633
Min similarity in groups: 0.600 (Threshold)
Max similarity in groups: 0.816
Exact matches (Actual == Predicted): 93 / 3885
It can be observed that the candidates are distributed much more evenly across the corpus, but the number of candidates in a group decreases dramatically. The number of exact matches is virtually unchanged (even slightly increased).
Conclusion: It appears that certain sections of the VMS bear a greater resemblance to other lines than others. This is not a coincidence, but indicates a structure within the text.