• Options
• Login
• Desktop Style

Split: Entropy digression

• Index

  • The Voynich Ninja
  • Voynich Research
  • Analysis of the text
  • Split: Entropy digression
• Split: Entropy digression

• Split: Entropy digression

  ReneZ > 08-07-2026, 01:08 AM

  (07-07-2026, 09:39 PM)Stefan Wirtz_2 Wrote: You are not allowed to view links. Register or Login to view.Those entropy calculations are still very depending upon the type of transliteration;
  some more aggressive "experts" here are favourizing an "h2" of 2.3 for Voynichese, while the change of transliteration file can produce a value of ~2.9.

  
  As @nablator already pointed out, this is not correct. You wrote that this value was reported for the v101 transliteration. I just recomputed this with my own tools, stripping the transliteration of all annotations, but keeping the full character set.
  
  The v101 file uses 158 different characters, if one counts the space also as a character.
   
  Computing the entropy values, also counting space as a character yields:
  Single char entropy     H1 =  4.036
  Character pair entropy H2 =  6.608
  Conditional ch entropy h2 =   2.572
  
  More realistic is not to count space as a character (it isn't). Then:
  Single char entropy     H1 =  4.131
  Character pair entropy H2 =  6.501
  Conditional ch entropy h2 =   2.370
  
  In this case, character pairs spanning a space are not included in the statistics.
  
  The most representative value is 2.37.
  
  Comparison values may be found here: You are not allowed to view links. Register or Login to view.
  and here: You are not allowed to view links. Register or Login to view.

• RE: Cvetka's Slovenian Theory Thread

  Stefan Wirtz_2 > Yesterday, 02:34 PM

  (08-07-2026, 01:08 AM)ReneZ Wrote: You are not allowed to view links. Register or Login to view.As @nablator already pointed out, this is not correct. You wrote that this value was reported for the v101 transliteration. I just recomputed this with my own tools, stripping the transliteration of all annotations, but keeping the full character set.

  
  As I pointed out, I do not do the calculations and used Ed's tool. This one is still producing values between 2.3 and 2.89... for whole VMS corpus, just by choosing different transliterations. 
  I have no reason to doubt these numbers and formulas behind it.
  
  
  

  (08-07-2026, 01:08 AM)ReneZ Wrote: You are not allowed to view links. Register or Login to view.The v101 file uses 158 different characters, if one counts the space also as a character.
   
  Computing the entropy values, also counting space as a character yields:
  Single char entropy     H1 =  4.036
  Character pair entropy H2 =  6.608
  Conditional ch entropy h2 =   2.572
  
  More realistic is not to count space as a character (it isn't). Then:
  Single char entropy     H1 =  4.131
  Character pair entropy H2 =  6.501
  Conditional ch entropy h2 =   2.370
  
  In this case, character pairs spanning a space are not included in the statistics.
  
  The most representative value is 2.37.

  
  Should be normal not to see 'space' as a character, but do "your tools" agree with that?
  You cited @nablator, so I checked his tool at nablator.free.fr :
  
  
  
  Funny things happen when you just insert a few more spaces there, or even 'line feed'/'carriage return' (better known as 'return' key) signals into that: h2 increases, and drops down after ~3rd input.
  Saw this when I inserted my own transliteration with much return characters to keep the page format.
  
  You remarked the 'spaces' already, but a tool that is taking 'returns' as symbols? Come on, did you even check your own tools...?
  
  Apart from that: I reviewed Bowern/Lindemann the last days; the used 300 living languages (leaving out just 6.700 that exist(ed) actually) and made comparisons with their today's Wikipedia corpuses! 
  Surely, h2 etc. are much better by this, but:
  - how many old songs are in todays Wikipedia texts?
  - how many appraisals or prayers?
  - how many tales and legends?
  - how many full Bible versions or other religious basic texts?
  
  B/L also took historical texts for their studies, but that data base is significantly lower and has big geographical gaps.
  
  Finally, the question about low entropies should get answered with all possible reasons to have such; 
  here, it is nearly only the "consensus" that VMS must be enciphered somehow.
  And all "substitution" versions are declared as "impossible" a priori -- which was never proven by entropy calculations itself, but by those "solvers" who failed with substitutions into wrong languages / alphabets by bad choices.
  
  So if it is a cipher: just decipher it, can't be too hard with a cipher that produces 3-4 identical Vords in a row...

• Entropy digression

  ReneZ > Today, 01:17 AM

  (Yesterday, 02:34 PM)Stefan Wirtz_2 Wrote: You are not allowed to view links. Register or Login to view.You remarked the 'spaces' already, but a tool that is taking 'returns' as symbols? Come on, did you even check your own tools...?

  
  Of course, come on!
  
  But did you check Ed's tool, which you seem to trust?
  
  As I clearly wrote, I computed the entropy both 'with' the space as a character and 'without'.
  
  All other meaningless characters (returns, commas, repeated spaces, whatever) are ignored (of course!).
  May I suggest that it is a bit odd that you need to play with that, in order to get alternative values out?
  I can decide to include or exclude numbers as valid characters, or to treat upper case or lower case as either different or the same, among other things. 
  
  For the v101 transliteration, I need to keep the numbers in, also distinguish between upper and lower case, and a special complication is that the high-ascii codes are conflicting with standard unicode usage, so I had to convert these to UTF-8 two-byte codes. Of course, I checked that these were properly recognised when computing the statistics. Just to be clear: preparing the text took more time and effort than actually computing the numbers.
  
  So please refrain from confusing people by repeatedly presenting invalid numbers that have already been pointed out by two independent and coinciding results.
  
  The ball is now in your court to double-check that the tool you used properly handled the above-mentioned points.
  Or you can also just believe us.

• RE: Entropy digression

  Stefan Wirtz_2 > 5 hours ago

  (Today, 01:17 AM)ReneZ Wrote: You are not allowed to view links. Register or Login to view.Of course, come on!
  
  But did you check Ed's tool, which you seem to trust?
  As I clearly wrote, I computed the entropy both 'with' the space as a character and 'without'.
  All other meaningless characters (returns, commas, repeated spaces, whatever) are ignored (of course!).
   

  
  Ed's tool seems to use the spaces to seperate 'Vords' as tokens; you can't really "ignore" spaces and need them, but just better not see them as characters on their own, do you?
  @nablator's online tool is reacting to both, spaces and returns, which can change results unknowingly to the user, depending on qualitiy of inserted texts. Not the best base for "being sure about entropy".
  
  

  (Today, 01:17 AM)ReneZ Wrote: You are not allowed to view links. Register or Login to view.May I suggest that it is a bit odd that you need to play with that, in order to get alternative values out?

  
  May I suggest that it is a bit odd to critizise a few random tests of tools producing "results"?
  I don't know what tools you are using -- you did not say that.
  
  But quite obviously each and every tool is producing different entropy values, at least a bit. And that is additionally depending on text and transliteration quality a lot.
  
  

  (Today, 01:17 AM)ReneZ Wrote: You are not allowed to view links. Register or Login to view.I can decide to include or exclude numbers as valid characters, or to treat upper case or lower case as either different or the same, among other things. 
  For the v101 transliteration, I need to keep the numbers in, also distinguish between upper and lower case, and a special complication is that the high-ascii codes are conflicting with standard unicode usage, so I had to convert these to UTF-8 two-byte codes. Of course, I checked that these were properly recognised when computing the statistics. Just to be clear: preparing the text took more time and effort than actually computing the numbers.

  
  This sounds like a test of v101 was "too complicated" for your tools.
  Just to be clear: did you or did you not run a Glen Claston variant through your calculations? And what was the result of it?
  v101's character set may be much too high; but that numbers of ZL transliterations look rather low
  
  

  (Today, 01:17 AM)ReneZ Wrote: You are not allowed to view links. Register or Login to view.So please refrain from confusing people by repeatedly presenting invalid numbers that have already been pointed out by two independent and coinciding results.

  
  Please do not confuse people by repeatedly doubting (only) the highest h2 result from Ed's online tool, while you have not found any reasons or bugs within that calculation yet. At the same time, at least @nablators thing is a bit weird-working. So what's your point with "2 independent coinciding results"?
  
  

  (Today, 01:17 AM)ReneZ Wrote: You are not allowed to view links. Register or Login to view.The ball is now in your court to double-check that the tool you used properly handled the above-mentioned points.

  
  Is it?
  Obviously, I don't care much for entropies. 
  The "tool I mentioned" is accepting txt-files which I can change, and offers different options for the choice of transliterations. Quite comfortable, beyond that I have or want no further insight or access to it's functions. 
  But it produces constant results, not caring for line feed characters.
  
  So what's the point here?
  
  

  (Today, 01:17 AM)ReneZ Wrote: You are not allowed to view links. Register or Login to view.Or you can also just believe us.

  
  Aaah, I see. 
  Who is "us"?
  Independent from that, I will never just "believe" anything from anyone in a (sometimes) scientifical environment, only by some "we say so".
  
  To keep it a bit more rational:
  Yes, (h2) entropy values are very low under most circumstances. I wouldn't disagree.
  No, it is not always "too low".
  
  Yes, low entropies are strange in comparison to most modern languages and several known medieval texts. I wouldn't disagree. 
  No, this does not prove VMS being not in a natural language or not a 'normal' text, or being definitely a cipher.
  
  As long as those "coinciding results" exclusive lead to the claim "must be a cipher!" and are not regarding all other possible reasons for low entropies even in a long text, this is not a science-based discussion -- just an exchange of single opinions with more-or-less air superiority, building up majorities.
  
  "Structure" (as far as it can be recgnized) and some strange features can be found in european languages also. Beyond entropy acrobatics, this is nearly not regarded systematically yet.

• RE: Split: Entropy digression

  DG97EEB > 5 hours ago

  Dunsel also created one You are not allowed to view links. Register or Login to view.
  

  ---

  Dunsel also created one You are not allowed to view links. Register or Login to view.
  
  And as I said above, if you spot an issue with mine, tell me. These are running deterministic python in the background. But like any software, it's entirely possible there's a bug in one of the scripts.. I will take another look...

• RE: Split: Entropy digression

  eggyk > 1 hour ago

  (Today, 01:17 AM)ReneZ Wrote: You are not allowed to view links. Register or Login to view.The ball is now in your court to double-check that the tool you used properly handled the above-mentioned points.

  
  In cases of differing results to the norm, isn't the normal procedure to prove those results are invalid? It's a little unusual to go down the route of "your figures are wrong, prove to me that they are right". 
  
  

  (Today, 01:17 AM)ReneZ Wrote: You are not allowed to view links. Register or Login to view.Or you can also just believe us.

  
  This isn't productive way to discuss this, imo.
  
  It's entirely possible that the 2 tools you talked about both contain some systemic error, and that the 3rd does not. 2 sets of results is far from a consensus too, but even if it were, consensuses quite often can be wrong. (im not saying this is the case, but it's not impossible that it is the case).
• Next Oldest Next Newest