How to recombine glyphs to increase character entropy?

[-JKP-]

This snippet illustrates length variation in medieval Latin words (this is slightly on the long side, but it is not an extreme example, it's a reasonably normal one):


Many of the words in this selection are abbreviated, which shortens them, and yet even with the abbreviations, we see the following word lengths:

     6 10 15 5 11 3 8 3 3 4 10 5 9 9 3 6 9 4 4 (last word breaks across the line and indulgere breaks across the line)

• 19 words, 11 of them abbreviated (more than half)
  
• range: 3 to 15 characters (the last word has been chopped across a line, it is 4 chars, and indulgere is one word)
  
• average word length (unexpanded) 6.7 characters
  
• average word length (abbreviations expanded) 7.6 characters
  
• 16 different characters at the beginnings of words
  

   


Then I took a snippet from You are not allowed to view links. Register or Login to view. where some of the tokens tend to be longer. These stats are for the first 19 tokens:

    5 5 6 4 6 6 5 4 2 4 3 4 7 6 5 4 7 4 5

• 19 words (starting with tedal), unknown whether there are abbreviations
  
• range: 2 to 7 glyphs (and I was being generous in assigning "2" since it is a half-space, it may actually be 3 to 7 glyphs)
  
• average token length 4.8 (minims were treated as individual glyphs but they may not be)
  
• 6 different characters at the beginnings of tokens
  

   


But... the almost 30% difference in token length isn't the most important difference.

The more important patterns are:

• the variation in length, with medieval Latin ranging from 3 to 15 characters and the VMS within a narrower range, from 2 to 7 (Note that the "2" in this equation is actually a half-space. It might actually be a 6 rather than 2 + 4, which makes it even less varied than the Latin sample, but there are 2-glyph tokens in the VMS, so I decided to go with the half-space.)
  
• the positionality of specific characters in the VMS, which is partly illustrated by the limited character set used at the beginnings of tokens
  

If you take the spaces at face value, the VMS is lacking in word-length variation (in addition to being highly repetitious and positional).
If you are suspicious of spaces (I have been suspicious of spaces for a long time, partly because of this kind of pattern), then the possibility exists of abbreviations or markers/modifiers or both or tokens split across a space.

------------------
Caveat: I am well aware that one should not do statistical comparisons on small sample sets. In fact, it is folly to do so. But we have many statistical attacks on larger sample sets (and they are the ones that should be considered important) yet I still get the feeling that the differences are not being internalized by a number of people offering substitution solutions, and I still sense resistance to the idea that the spaces might be suspect.

So, I thought I would post this to help researchers visualize what is going on in the larger-sample studies but with a smaller amount of data and actual clips of text, just to get the idea across.