09-07-2026, 02:34 PM
(08-07-2026, 01:08 AM)ReneZ wrote: You are not allowed to view links. Register or Login to view.As @nablator already pointed out, this is not correct. You wrote that this value was reported for the v101 transliteration. I just recomputed this with my own tools, stripping the transliteration of all annotations, but keeping the full character set.
As I pointed out, I do not do the calculations and used Ed's tool. This one is still producing values between 2.3 and 2.89... for whole VMS corpus, just by choosing different transliterations.
I have no reason to doubt these numbers and formulas behind it.
(08-07-2026, 01:08 AM)ReneZ wrote: You are not allowed to view links. Register or Login to view.The v101 file uses 158 different characters, if one counts the space also as a character.
Computing the entropy values, also counting space as a character yields:
Single char entropy H1 = 4.036
Character pair entropy H2 = 6.608
Conditional ch entropy h2 = 2.572
More realistic is not to count space as a character (it isn't). Then:
Single char entropy H1 = 4.131
Character pair entropy H2 = 6.501
Conditional ch entropy h2 = 2.370
In this case, character pairs spanning a space are not included in the statistics.
The most representative value is 2.37.
Should be normal not to see 'space' as a character, but do "your tools" agree with that?
You cited @nablator, so I checked his tool at nablator.free.fr :
Funny things happen when you just insert a few more spaces there, or even 'line feed'/'carriage return' (better known as 'return' key) signals into that: h2 increases, and drops down after ~3rd input.
Saw this when I inserted my own transliteration with much return characters to keep the page format.
You remarked the 'spaces' already, but a tool that is taking 'returns' as symbols? Come on, did you even check your own tools...?
Apart from that: I reviewed Bowern/Lindemann the last days; the used 300 living languages (leaving out just 6.700 that exist(ed) actually) and made comparisons with their today's Wikipedia corpuses!
Surely, h2 etc. are much better by this, but:
- how many old songs are in todays Wikipedia texts?
- how many appraisals or prayers?
- how many tales and legends?
- how many full Bible versions or other religious basic texts?
B/L also took historical texts for their studies, but that data base is significantly lower and has big geographical gaps.
Finally, the question about low entropies should get answered with all possible reasons to have such;
here, it is nearly only the "consensus" that VMS must be enciphered somehow.
And all "substitution" versions are declared as "impossible" a priori -- which was never proven by entropy calculations itself, but by those "solvers" who failed with substitutions into wrong languages / alphabets by bad choices.
So if it is a cipher: just decipher it, can't be too hard with a cipher that produces 3-4 identical Vords in a row...