How to recombine glyphs to increase character entropy?

[Koen G]

After cleaning up my corpus to remove excess tabs, carriage returns, spaces etc, I established the following target values. The lowest value is the non-VM text with the lowest conditional entropy (h2) or h2/h1 ratio. The median value obviously depends entirely on the corpus, but it gives a vague indication of an average. I included only latin alphabet because there are some issues with other alphabets I'd like to clear out first.

- With spaces:

VM h2: 1.78 - 2.23
Lowest h2: 2.82
Median h2: 3.15

VM h2/h1: 0.47 - 0.57
Lowest h2/h1: 0.68
Median h2/h1: 0.77

- Spaces removed:

VM h2: 1.97 - 2.38
Lowest h2: 3.02
Median h2: 3.42

VM h2/h1: 0.52 - 0.62
Lowest h2/h1: 0.72
Median h2/h1: 0.82


Apart from the target values, the main takeaway so far is that removing spaces generally increases entropy. More in-depth analyses follow.

Another thing I should add is that when looking at the difference between h2 with and without spaces, the VM is in the first half. This means that most texts actually gain more character entropy than the VM does when removing spaces. My intuition would expect the opposite.

However, this is without modifications to EVA. I predict the VM will rise significantly in the "difference" column once EVA is modified.