Levenshtein distance line by line

[bi3mw]

I have experimentally measured the Levenshtein distance between the first and last word of each line.

My hope was to see a pattern, but it didn't turn out that way. Values below 3 are rather rare and they appear random.

[bi3mw]

I forgot to mention:
The shuffling is performed by Fisher-Yates's algorithm, also known as Knuth's shuffle algorithm. So here the words are very "randomly" re-sorted ( across line boundaries ).

When checking several folios, it is noticeable that very short words are often written at the end of the line which look like "appended". Of course this has an influence on the results.

[RobGea]

When checking several folios, it is noticeable that very short words are often written at the end of the line which look like "appended". Of course this has an influence on the results.

Currier noted a similar thing :

"The ends of the lines contain what seems to be, in many cases, meaningless symbols: little groups of letters which don't occur anywhere else,
and just look as if they were added to fill out the line to the margin. "

Though there seems to be some debate as to exactly what Currier meant.

[bi3mw]

I wouldn't be surprised if Currier was referring to the phenomenon I described. One could almost think that there is some kind of obfuscation ( for whatever ).

What I can say after my tests is that very short words probably are not that often, not even slightly modified, at the beginning of a line. This would have been noticeable in the results, at least in the folios I tested. For example, in folio 115v, words of length one to three letters occur zero times at the beginning of the line but nine times at the end of the line.

As said, a value below 3 is rare and with similarly short word pairs there are only a few possibilities of substitution.


edit: in the entire VMS, there are much more often short words at the end of a line as at the beginning ( Word length one to two letters, result 153 / 537 ).

Here are the word lists (one to two letters). I have removed all lines in the VMS corpus with only one word (for Levenshtein test). So the result is only approximate.

[MarcoP]

Hi Matthias,
this is an interesting subject that definitely deserves more investigation!
Something partially related was discussed You are not allowed to view links. Register or Login to view.. See also the previous posts in that thread and Elmar Vogt's paper about You are not allowed to view links. Register or Login to view..
It seems that words at the end of lines tend to be shorter also in normal printed texts. I doubt that this in an effect of hyphenation: short words are more likely to be squeezed at the end of a line, where there would not be room for a longer word, but I don't think I ever checked if this assumption is correct.

[-JKP-]

...
It seems that words at the end of lines tend to be shorter also in normal printed texts. I doubt that this in an effect of hyphenation: short words are more likely to be squeezed at the end of a line, where there would not be room for a longer word, but I don't think I ever checked if this assumption is correct.

I had never really thought about this, but you're right. Short words are more likely to be added to the end of the line where there is space for a short word but not for a long one.

In medieval texts, this might be balanced somewhat by words that are continued on the next line, but not every scribe broke words across lines.

[bi3mw]

It seems that words at the end of lines tend to be shorter also in normal printed texts.

I'm not sure about that, but in the VMS there are noticeably often short words at the end of the line. It often seems as if they were appended to create a homogeneous line spacing to the right margin. So they are possibly filler words. This possibility would, as said, have a strong influence on the result and is discussed further You are not allowed to view links. Register or Login to view. .