The Voynich Ninja

(07-07-2026, 09:39 PM)Stefan Wirtz_2 Wrote: You are not allowed to view links. Register or Login to view.Those entropy calculations are still very depending upon the type of transliteration;
some more aggressive "experts" here are favourizing an "h2" of 2.3 for Voynichese, while the change of transliteration file can produce a value of ~2.9.

As @nablator already pointed out, this is not correct. You wrote that this value was reported for the v101 transliteration. I just recomputed this with my own tools, stripping the transliteration of all annotations, but keeping the full character set.

The v101 file uses 158 different characters, if one counts the space also as a character.
 
Computing the entropy values, also counting space as a character yields:
Single char entropy     H1 =  4.036
Character pair entropy H2 =  6.608
Conditional ch entropy h2 =   2.572

More realistic is not to count space as a character (it isn't). Then:
Single char entropy     H1 =  4.131
Character pair entropy H2 =  6.501
Conditional ch entropy h2 =   2.370

In this case, character pairs spanning a space are not included in the statistics.

The most representative value is 2.37.

Comparison values may be found here: You are not allowed to view links. Register or Login to view.
and here: You are not allowed to view links. Register or Login to view.

(08-07-2026, 01:08 AM)ReneZ Wrote: You are not allowed to view links. Register or Login to view.As @nablator already pointed out, this is not correct. You wrote that this value was reported for the v101 transliteration. I just recomputed this with my own tools, stripping the transliteration of all annotations, but keeping the full character set.

As I pointed out, I do not do the calculations and used Ed's tool. This one is still producing values between 2.3 and 2.89... for whole VMS corpus, just by choosing different transliterations. 
I have no reason to doubt these numbers and formulas behind it.

(08-07-2026, 01:08 AM)ReneZ Wrote: You are not allowed to view links. Register or Login to view.The v101 file uses 158 different characters, if one counts the space also as a character.
 
Computing the entropy values, also counting space as a character yields:
Single char entropy     H1 =  4.036
Character pair entropy H2 =  6.608
Conditional ch entropy h2 =   2.572

More realistic is not to count space as a character (it isn't). Then:
Single char entropy     H1 =  4.131
Character pair entropy H2 =  6.501
Conditional ch entropy h2 =   2.370

In this case, character pairs spanning a space are not included in the statistics.

The most representative value is 2.37.

Should be normal not to see 'space' as a character, but do "your tools" agree with that?
You cited @nablator, so I checked his tool at nablator.free.fr :



Funny things happen when you just insert a few more spaces there, or even 'line feed'/'carriage return' (better known as 'return' key) signals into that: h2 increases, and drops down after ~3rd input.
Saw this when I inserted my own transliteration with much return characters to keep the page format.

You remarked the 'spaces' already, but a tool that is taking 'returns' as symbols? Come on, did you even check your own tools...?

Apart from that: I reviewed Bowern/Lindemann the last days; the used 300 living languages (leaving out just 6.700 that exist(ed) actually) and made comparisons with their today's Wikipedia corpuses! 
Surely, h2 etc. are much better by this, but:
- how many old songs are in todays Wikipedia texts?
- how many appraisals or prayers?
- how many tales and legends?
- how many full Bible versions or other religious basic texts?

B/L also took historical texts for their studies, but that data base is significantly lower and has big geographical gaps.

Finally, the question about low entropies should get answered with all possible reasons to have such; 
here, it is nearly only the "consensus" that VMS must be enciphered somehow.
And all "substitution" versions are declared as "impossible" a priori -- which was never proven by entropy calculations itself, but by those "solvers" who failed with substitutions into wrong languages / alphabets by bad choices.

So if it is a cipher: just decipher it, can't be too hard with a cipher that produces 3-4 identical Vords in a row...

(09-07-2026, 02:34 PM)Stefan Wirtz_2 Wrote: You are not allowed to view links. Register or Login to view.You remarked the 'spaces' already, but a tool that is taking 'returns' as symbols? Come on, did you even check your own tools...?

Of course, come on!

But did you check Ed's tool, which you seem to trust?

As I clearly wrote, I computed the entropy both 'with' the space as a character and 'without'.

All other meaningless characters (returns, commas, repeated spaces, whatever) are ignored (of course!).
May I suggest that it is a bit odd that you need to play with that, in order to get alternative values out?
I can decide to include or exclude numbers as valid characters, or to treat upper case or lower case as either different or the same, among other things. 

For the v101 transliteration, I need to keep the numbers in, also distinguish between upper and lower case, and a special complication is that the high-ascii codes are conflicting with standard unicode usage, so I had to convert these to UTF-8 two-byte codes. Of course, I checked that these were properly recognised when computing the statistics. Just to be clear: preparing the text took more time and effort than actually computing the numbers.

So please refrain from confusing people by repeatedly presenting invalid numbers that have already been pointed out by two independent and coinciding results.

The ball is now in your court to double-check that the tool you used properly handled the above-mentioned points.
Or you can also just believe us.

(10-07-2026, 01:17 AM)ReneZ Wrote: You are not allowed to view links. Register or Login to view.Of course, come on!

But did you check Ed's tool, which you seem to trust?
As I clearly wrote, I computed the entropy both 'with' the space as a character and 'without'.
All other meaningless characters (returns, commas, repeated spaces, whatever) are ignored (of course!).
 

Ed's tool seems to use the spaces to seperate 'Vords' as tokens; you can't really "ignore" spaces and need them, but just better not see them as characters on their own, do you?
@nablator's online tool is reacting to both, spaces and returns, which can change results unknowingly to the user, depending on qualitiy of inserted texts. Not the best base for "being sure about entropy".

(10-07-2026, 01:17 AM)ReneZ Wrote: You are not allowed to view links. Register or Login to view.May I suggest that it is a bit odd that you need to play with that, in order to get alternative values out?

May I suggest that it is a bit odd to critizise a few random tests of tools producing "results"?
I don't know what tools you are using -- you did not say that.

But quite obviously each and every tool is producing different entropy values, at least a bit. And that is additionally depending on text and transliteration quality a lot.

(10-07-2026, 01:17 AM)ReneZ Wrote: You are not allowed to view links. Register or Login to view.I can decide to include or exclude numbers as valid characters, or to treat upper case or lower case as either different or the same, among other things. 
For the v101 transliteration, I need to keep the numbers in, also distinguish between upper and lower case, and a special complication is that the high-ascii codes are conflicting with standard unicode usage, so I had to convert these to UTF-8 two-byte codes. Of course, I checked that these were properly recognised when computing the statistics. Just to be clear: preparing the text took more time and effort than actually computing the numbers.

This sounds like a test of v101 was "too complicated" for your tools.
Just to be clear: did you or did you not run a Glen Claston variant through your calculations? And what was the result of it?
v101's character set may be much too high; but that numbers of ZL transliterations look rather low

(10-07-2026, 01:17 AM)ReneZ Wrote: You are not allowed to view links. Register or Login to view.So please refrain from confusing people by repeatedly presenting invalid numbers that have already been pointed out by two independent and coinciding results.

Please do not confuse people by repeatedly doubting (only) the highest h2 result from Ed's online tool, while you have not found any reasons or bugs within that calculation yet. At the same time, at least @nablators thing is a bit weird-working. So what's your point with "2 independent coinciding results"?

(10-07-2026, 01:17 AM)ReneZ Wrote: You are not allowed to view links. Register or Login to view.The ball is now in your court to double-check that the tool you used properly handled the above-mentioned points.

Is it?
Obviously, I don't care much for entropies. 
The "tool I mentioned" is accepting txt-files which I can change, and offers different options for the choice of transliterations. Quite comfortable, beyond that I have or want no further insight or access to it's functions. 
But it produces constant results, not caring for line feed characters.

So what's the point here?

(10-07-2026, 01:17 AM)ReneZ Wrote: You are not allowed to view links. Register or Login to view.Or you can also just believe us.

Aaah, I see. 
Who is "us"?
Independent from that, I will never just "believe" anything from anyone in a (sometimes) scientifical environment, only by some "we say so".

To keep it a bit more rational:
Yes, (h2) entropy values are very low under most circumstances. I wouldn't disagree.
No, it is not always "too low".

Yes, low entropies are strange in comparison to most modern languages and several known medieval texts. I wouldn't disagree. 
No, this does not prove VMS being not in a natural language or not a 'normal' text, or being definitely a cipher.

As long as those "coinciding results" exclusive lead to the claim "must be a cipher!" and are not regarding all other possible reasons for low entropies even in a long text, this is not a science-based discussion -- just an exchange of single opinions with more-or-less air superiority, building up majorities.

"Structure" (as far as it can be recgnized) and some strange features can be found in european languages also. Beyond entropy acrobatics, this is nearly not regarded systematically yet.

Dunsel also created one You are not allowed to view links. Register or Login to view.

---

Dunsel also created one You are not allowed to view links. Register or Login to view.

And as I said above, if you spot an issue with mine, tell me. These are running deterministic python in the background. But like any software, it's entirely possible there's a bug in one of the scripts.. I will take another look...

(10-07-2026, 01:17 AM)ReneZ Wrote: You are not allowed to view links. Register or Login to view.The ball is now in your court to double-check that the tool you used properly handled the above-mentioned points.

In cases of differing results to the norm, isn't the normal procedure to prove those results are invalid? It's a little unusual to go down the route of "your figures are wrong, prove to me that they are right". 

(10-07-2026, 01:17 AM)ReneZ Wrote: You are not allowed to view links. Register or Login to view.Or you can also just believe us.

This isn't productive way to discuss this, imo.

It's entirely possible that the 2 tools you talked about both contain some systemic error, and that the 3rd does not. 2 sets of results is far from a consensus too, but even if it were, consensuses quite often can be wrong. (im not saying this is the case, but it's not impossible that it is the case).

I've audited the numbers against Rene's site and they are correct. I have added some additional ones as pointed out by Rene specifically, so you can trace every one..

We are not talking about opinions here, but about straightfoward numerical calculations.
About facts, really.

There are more people than just nablator and myself who can do entropy calculations.

(10-07-2026, 05:35 PM)eggyk Wrote: You are not allowed to view links. Register or Login to view.It's a little unusual to go down the route of "your figures are wrong, prove to me that they are right". 

It is entirely resaonable, when two independent alternative results are consistent, and contradict a third one, that the third one goes back to check what he's done. In fact, refusing to do so is hardly scientific, and certainly not good (software) engineering practice.

@nablator already pointed out, after the first (of now three) times that Stefan Wirtz produced his incorrect values, that he probably processed the v101 transliteration without removing the annotations in the file, like:

<67v2.label_upper_left>
<67v2.label_upper_right>

I don't know if that's what happened but it is a very reasonable suggestion. It would also not be a fault of any software, but of the person running the test. The ball remains squarely in Stefan Wirtz' court. 

(10-07-2026, 05:35 PM)eggyk Wrote: You are not allowed to view links. Register or Login to view.

Quote:Or you can also just believe us.

This isn't productive way to discuss this, imo.

It was just presented as an option :-)
In fact, I would also consider it good advice.

EDIT:
just for completeness, let me add a You are not allowed to view links. Register or Login to view., which is very clear and very specific.

He shows how the incorrect value of Stefan Wirtz (2.87 rounded to 2.9) can be obtained, and how the correct calculation leads to 2.59 when one includes spaces as characters. My value was 2.57 with spaces and 2.37 without. We may not have been processing uncertain spaces in the same way.
Not treating spaces as characters is more useful, but what matters is that one does the same for the Voynich text and the comparison texts.

(10-07-2026, 11:53 PM)ReneZ Wrote: You are not allowed to view links. Register or Login to view.My value was 2.57 with spaces and 2.37 without. We may not have been processing uncertain spaces in the same way.

Without formatting, with all spaces kept (dots and commas converted to spaces) I get 2.58. Close enough. (My web app also converts ends of lines to spaces so there is no need to replace them in voyn_101_bare_UTF8.txt.)

(10-07-2026, 11:53 PM)ReneZ Wrote: You are not allowed to view links. Register or Login to view.The v101 file uses 158 different characters, if one counts the space also as a character.

I counted 171 different characters with Notepad++ in voyn_101_bare_UTF8.txt. My web app agrees: h0 = log2(171).
[attachment=16475]

Without spaces there is a big discrepancy in h2. I removed all formatting including spaces (dots and commas) and line feeds, converted to UTF-8. (My web app expects UTF-8 when loading drag & dropped files.)

I have:
h1 = 4.181
h2 = 2.954

Can you please check?
[attachment=16474]

You are not allowed to view links. Register or Login to view.

(11-07-2026, 10:05 AM)nablator Wrote: You are not allowed to view links. Register or Login to view.

(10-07-2026, 11:53 PM)ReneZ Wrote: You are not allowed to view links. Register or Login to view.My value was 2.57 with spaces and 2.37 without. We may not have been processing uncertain spaces in the same way.

Without formatting, with all spaces kept (dots and commas converted to spaces) I get 2.58. Close enough. (My web app also converts ends of lines to spaces.)

(10-07-2026, 11:53 PM)ReneZ Wrote: You are not allowed to view links. Register or Login to view.The v101 file uses 158 different characters, if one counts the space also as a character.

I counted 171 different characters with Notepad++ in voyn_101_bare_UTF8.txt. My web app agrees: h0 = log2(171).



Without spaces there is a big discrepancy. I removed all formatting including spaces (dots and commas) and line feeds, converted to UTF-8. (My web app expects UTF-8 when loading drag & dropped files.)

I have:
h1 = 4.181
h2 = 2.954

You are not allowed to view links. Register or Login to view.

Can you please check?

I get the same... I've updated the site so you can include or exclude space and change boundaries...

I've also added an MCP so you can call it from Claude directly, which solves the issue of LLMs mangling calculations You are not allowed to view links. Register or Login to view.