26-05-2026, 08:17 AM
So is the low entropy of the VMS partly due to a segmentation effect?
I am referring here to the entropy of the glyph or symbol stream, specifically the conditional entropy H(next | current), not to word entropy. Word entropy would require reliable word boundaries, and these have not yet been definitively established here.
I am aware that resegmentation can generally increase entropy. But that is not the point. If certain vowel bigrams and consonant clusters function as merged units, then their internal glyph transitions will naturally be highly predictable.
If these units are then measured as ordinary consecutive glyphs, their internal predictability is counted as part of the text’s entropy. This would artificially increase predictability and correspondingly lower the measured conditional entropy at the glyph level.
In my tests, this new structural segmentation appears to raise the entropy toward the range expected for a normal language-based stream. This is not easy to compare directly, because the MHD comparison material has to be adjusted to the same structural level. But roughly speaking, the direction is correct - and it is also logically what one would expect.
I am referring here to the entropy of the glyph or symbol stream, specifically the conditional entropy H(next | current), not to word entropy. Word entropy would require reliable word boundaries, and these have not yet been definitively established here.
I am aware that resegmentation can generally increase entropy. But that is not the point. If certain vowel bigrams and consonant clusters function as merged units, then their internal glyph transitions will naturally be highly predictable.
If these units are then measured as ordinary consecutive glyphs, their internal predictability is counted as part of the text’s entropy. This would artificially increase predictability and correspondingly lower the measured conditional entropy at the glyph level.
In my tests, this new structural segmentation appears to raise the entropy toward the range expected for a normal language-based stream. This is not easy to compare directly, because the MHD comparison material has to be adjusted to the same structural level. But roughly speaking, the direction is correct - and it is also logically what one would expect.