An Essay on Entropy: what is it, and why is it so important?

[kckluge]

Please post more, Karl!

From a priorities perspective, I shouldn't...but I probably will...but I really shouldn't...

A couple quick (and thread-relevant) things:

* With regard to the whole are "words" _words_ issue -- I've seen lots of papers talk about Voynich "words" and Zipf's (word frequency vs word frequency rank) Law. There is a different Zip's Law relating word length vs word frequency rank -- more frequent words are also shorter (a ref for anyone wanting to pull on that thread is You are not allowed to view links. Register or Login to view.) -- has anyone every fit that vs. the Voynich vocabulary? Based on the 32 most-common BioB "words" from D'Imperio's transcription:

33    OFCC89        33    ZCC89        34      SCC9        35      2AM
  35      SX9        40    4OFC9        40  4OPCC89        41      OFAN
  43      8AN        44        89        44    OFC89        46    4OPC89
  47    4OFAR        50    OPC89        53      8AE        56      8AR
  56    ESC89        57        OR        58      4OF9        75      SC9
  77      8AM        78      ZC9        85    4OFAM        85    4OFCC9
  87      4OE        106    4OFAE        150    4OFAN        152  4OFCC89
161    4OFC89        178        OE        194      SC89        213      ZC89


I'm not sure how well that will fit. Recognizing that word length is a function of transcription alphabet, if it hasn't been done that would be an interesting thing to check.

* With regard to verbose ciphers, the way Koen put things in his thread starter ("A verbose cipher is basically a cipher that obfuscates by adding unnecessary stuff. In a very simple example, I could verbosely obfuscate the word "Voynich" by adding a v after evert letter: "Vvovyvnvivcvhv". If Voynichese is the result of a verbose cipher, I could try to reverse this by rewriting common glyph clusters (bigrams, trigrams) as single glyphs. For example, I could replace "dy" by "&" and run the entropy test again to see what changed.") kind of half brings out a distinction between two different types of verbose cipher theories:

In the first type, like with his "Vvovyvnvivcvhv" example, nulls are being inserted in a structured way. In this type, it might be the case that some Currier P's have O's in front of them not because OP represents a different plaintext letter than P, but because there is some rule that puts O's in as nulls before some P's. Without "showing my work", if you take Herbal A and:

replace "4O" with 'o'
replace "CC" with 'c'
then get rid of A's before [ERNM], O's before [ERFP], and C's before [89]

you also wind up with a transformed text whose h1 and h2 are getting more natural language-like. Having looked for common strings shared between pages I don't know that it's a very promising possibility, but it is one.

The second type is where some plaintext letters map to a single glyphs, while some map to 2-(or 3- or more glyph combos -- so plaintext 'C' might map to Currier "OP", while plaintext 'K' might map to Currier "P".  A straddling checkerboard is a (from the point of view of the C-14 date somewhat anachronistic) cipher of this type (You are not allowed to view links. Register or Login to view. shows examples). The problem this raises is that if you have "CLOCK" as a plaintext word, then the "CK" at the end would result in "OPP" in the Voynich text given the hypothetical mappings -- and except for Currier 'C', repeated glyphs vanishingly rare -- out of 35483 digrams in (D'Imperio's) BioB, there are 15 that are doubled glyphs other than "CC". So either there just miraculously happens to be a natural language whose letter contact stats allow a mapping that avoids that problem, or there is some mechanism in the cipher that hides such doubled glyphs -- perhaps the difference between Currier "OF" and "OP" isn't that they map to different plaintext letters, perhaps "OF" is what you write where "OPP" would otherwise occur. Or maybe the verbose cipher theory is just fundamentally wrong...