Words around text intrusions/image breaks and towards the start/end of lines

[Jorge_Stolfi]

common thing in medieval times was to make lines very cleanly justified by any means necessary,

That is still true today.  Decent book publishers do not use line fillers and leave the tail line of paragraphs short.  But they will use hyphenation and tweak the spaces between words, sometimes even between letters, in order to get an even right rail (right edge of the text).

The VMS Scribe did not use hyphenation (that we know of), but it seems that he used abbreviations.  Many lines end in am, and m seems less common at other places along the line.

On the other hand, he was not overly concerned with appearance.  His lines are all bent and tilted and unevenly spaced, and the seems to have regarded the right rail like Pirate's Code: more like a suggestion than a binding rule...

I hypothesized that this would mess with statistics around drawings and line breaks. ... Words at line end and before text intrusions are shorter on average, by around 0.3 glyphs and 0.5 glyphs, respectively.

You don't need an obsessive-compulsive Scribe for that.  

The trivial line-breaking algorithm is: if there is enough space on the current line for the next word, write it and continue.  If not, break the line there and start a new line at the left rail.

People don't seem to realize, but this banal algorithm results in the first word of each line being longer than average, with the last 1-3 words of each line being shorter than average.  

If the first word of the line is longer, it means that the frequency of each word at the start of a line is different from its frequency at the end, and different from its frequency elsewhere along the line.  The word "therefore" will be more common than expected at the beginning of lines, while the words "the", "is", "if", "in" will be much more common at the end of lines.

If the word frequencies are different, then the frequencies of characters (and digraphs etc) will be different too.  Because the frequency of a character is determined mainly by whether it occurs in the most common words.  The digraph "th" is common in English because it occurs in many common words like "the", "this", "that", "they", "them", "their", "there", "thus", "then", "than"...  But these words are shorter than average, so they will be more common at the end of lines than at the beginning.  And thus the same will happen to the frequency of "th".

Thus anyone claiming that X or Y is more common than expected at the start or end of lines should check whether that anomaly is not caused by this effect.

The letter "e" is significantly less common in words around breaks ... The letter "s" is about 3 times as common in words at the start of lines and both before and after drawings, and 63% more common at the end of lines ... 

These anomalies could be due to the letter e being used mostly in medium-length words like  cheedy and used less in both longer and shorter words.

There are also a lot of short words that seem to almost exclusively appear in these positions, like "sy", "oly", "oldy", "dy", "oky", "ldy", "ary", "lol", etc. My guess is that these words are either abbreviations or nonsense to pad for length.

Those may be abbreviated words indeed.

This has been You are not allowed to view links. Register or Login to view., but words at the start of lines often have a "d" or "y" for padding if the words starts with "ch" or "sh", creating a lot of words that almost exclusively appear at line start.

It makes no sense to have padding at the start of a line.  It may be simply that many long words start with y or d.

The word "sho" being a strange exception, and also a word that appears primarily at the start of lines.

The Scribe was not very consistent about the width of spaces. He often left wider spaces between certain pairs of letters, such as between an o and a gallows.  Transcribers have a hard time deciding whether some gaps are word breaks or not. Thus those Sho at line start may have been part of a long word that got split in the transcription.

And I must insist on two pieces of advice:

1. Letter and digraph statistics are projections of word statistics.  Thus the frequency of "u" in English is a combination of the frequencies of the words that have "u", like "thus", "tetanus", "butane", "Zulu", and "weltanschauung".  For that reason, such statistics are often more confusing than illuminating.  It is like trying to study the ecology of a place by counting animals with no tails, like gorillas, toads, and cockroaches; with short tails, like turtles and elephants; and with long tails,  like lions and snakes.  From those numbers one may learn that the Sahara is different from the Amazon, but one would never understand why, or go beyond that.  Thus, whenever possible, compute and study the frequency of words rather than glyphs.
   
2. There is no such things as the frequency of a letter of word in a language. Statistics are properties of specific texts, not of their languages.  Different texts can naturally have very different frequencies, even for basic function words like "the", "and", "is", because their topic and style influences the frequency of words, sentence structures, tenses, etc..  Therefore, when computing such statistics, be careful to use a text that is as homogeneous as possible.  Do not lump labels with paragraph, Herbal with Bio, etc.  It is better to study just one section of the VMS in detail than try to understand all sections at the same time -- even if you compute the statistics separately for each section.
   

All the best, --stolfi