Bit-level Geometry: A deterministic discovery of Voynichese building blocks

[Mauro]

2. The raw, uncompressed EVA transcription is 1,862,208 bits or 232,776 bytes. It depends on which Lens you run through the engine (standard MDL formula) so here's each of them:

• 8-bit: 903,682 bits (~51.5% reduction)
  
• 16-bit: 723,282 bits (~61.2% reduction)
  
• 24-bit: 688,640 bits (~63.2% reduction) (winner)
  
• 32-bit: 782,379 bits (~57.9% reduction)
  

This is an intriguing result for me. 24-bit (3 characters) is the mathematical sweet spot which matches what Voynich researchers have long suspected. It suggests the text is built out of tight, highly constrained prefixes, roots and suffixes (e.g. "qok", "che", "dai").



Cheers!

I think it would be interesting to compare your results with mine.

I used a different approach, modeling the VMS word types using a slot grammar, which was then used to derive the 'chunks' composing the words. Then I applied a metric (which I called Nbits): the number of bits needed to compress the textg usin the identified chunks as the dictionary for the compression algorithm (you can find the procedure here: You are not allowed to view links. Register or Login to view. , the text was compressed using Hufmann codes).

The best compression I found (see You are not allowed to view links. Register or Login to view. and following posts, included a discussion of the problems I found) required 480985 bits to encode the whole EVA text (not guaranteed at all to be optimal in any sense, I used a Monte Carlo engine to sample different 'chunkifications' of words). 15314 bits for the dictionary, 465671 bits for the text. 

I looked forward to an algorithm implementing a 'chunkification ' of words without being supervised, so I'm happy to see yours, unsupervised and MLD-based (but I also agree with others here, it's not a given that this can be done meaningfully). It would be nice to compare the compression rate you achieved with mine, however, there are a lot of caveats in comparing the two results: I used Huffman codes to compress the text, I considered spaces as word separators (and now I'm unsure of how/if I counted them in the Nbits figure), I removed words with 'rare' characters (from EVA 'g' downwards). Not knowing exactly what you did, it's hard to say how to make the comparison (nor I have much time for it). What do you think? Does your 688640 bits improves on mine 480985 [I hope so]?

And, did you find many 'chunks' which include a space in the middle? Ie. like "y q"?