An Essay on Entropy: what is it, and why is it so important?

[Koen G]

Another way to think about abbreviation is that it will target areas of a text that are low-entropy to begin with. A common glyph cluster, a common ending. These things can be abbreviated because they are low entropy. Their predictability makes it so that I can still read the text if they are gone or replaced. 

The little bit I wrote about information density also comes into play here. When abbreviating, you are looking for those bits that are easy to predict and hence carry little crucial information. The fact that they can be shortened or omitted implies that they were too "verbose" to begin with. This is why abbreviation will increase the entropy of a text.

Note: the abbreviations I have in mind here are ones that could be used by medieval scribes. I am sure we could invent some novel form of abbreviation that does decrease entropy, but then I wonder of the text would still be legible, because you'd have to shorten a text and reduce its information density at the same time!

Conversely, if I "abbreviate" an EVA transcription of Voynichese by rewriting common glyph clusters as single glyphs, I will increase its entropy. So one could say that Voynichese needs to be abbreviated to bring it closer to a normal text, rather than the other way around. This, in turn, would mean that "words aren't words", because Voynichese words would become much too short, and we now shifted the problem from entropy to spaces.

Edit: oops, I was typing this reply while Marco made his post - seems like there is some overlap.