How to recombine glyphs to increase character entropy?

[MarcoP]

After discussing things and exchanging files with Koen via email, we might have understood the differences between our results. The main reason is that, when removing spaces, Koen also removed line separators (newlines), so that the text is transformed in a single very long word. In any text, removing spaces results in bigrams that typically do not occur. In an ordinary text, when removing newlines you get the same bigrams you generate when removing spaces. But the VMS has characters that are considerably more frequent at line start (e.g. f- p- ) or line end (e.g -m -g): when removing newlines you generate new bigrams like 'mf' 'mp'.
Another difference is that I used the Zandbergen Landini transcription, while Koen used Takahashi's. But I don't think this really makes a difference.

Here is an updated table, where I tried to more closely replicate Koen's way of processing the files:

            WITH_SPACES                        NO_SPACES/NEWLINES          DIFF
___FILE___ char.h bigr.h cond.h cond/char| char.h bigr.h cond.h cond/char| cond/char
VMS_TT_H.A 3.8178 5.9237 2.1059 0.5516   | 3.8107 6.1390 2.3283 0.6110   | 0.0594
VMS_TT_Q13 3.7949 5.6247 1.8298 0.4822   | 3.8026 5.7961 1.9935 0.5242   | 0.0420
Machiavell 3.9354 7.0337 3.0982 0.7873   | 3.9823 7.3056 3.3234 0.8345   | 0.0472
___Pliny__ 4.0127 7.3793 3.3666 0.8390   | 4.0025 7.5086 3.5061 0.8760   | 0.0370