- Options
- Login
- Desktop Style
Options
Question about unicity distance
- Question about unicity distance
-
Question about unicity distance
kckluge > 11 hours ago
I could swear there was a reference to unicity distance in a recent comment, and I was going to ask this there, but for the life of me I can't find it -- so here we go...
Unicity distance "is the length of an original ciphertext needed to break the cipher by reducing the number of possible spurious keys to zero in a brute force attack" (You are not allowed to view links. Register or Login to view.). It's a potential way of trying to fomalize the "too many degrees of freedom" critique of proposed Voynich solutions as well as other possible prunings (i.e., if you can't get a coherent stretch of plaintext from a 28+ character long stretch of the ciphertext you haven't proposed a credible solution of the text as a simple monoalphabetic cipher).
What I can't quite figure out is how to correctly compute the unicity distance for something like Brumbaugh's proposed cipher. To recap for those unfamiliar with his "solution", encipherment proceeded in two stages:
1) Convert from plaintext letters to digits using the following grid:
a, j, v = 1
b, k, r = 2
c, l, w = 3
d, m, s = 4
e, n, x = 5
f, o, t = 6
g, p, y = 7
h, q, u = 8
i, -us, z = 9
2) replace each digit with one of several glyphs corresponding to that digit
So, for instance, his hypothetical enciperhing of one of the labels near the upper right corner of You are not allowed to view links. Register or Login to view. is
P E P P E R --> 7 5 7 7 5 2 --> (EVA) s a r ch a r
so each of the three instances of the digit 7 gets replaced by a different Voynich script glyph ('s', 'r', and 'ch'), while both 5's get replaced by EVA 'a' (for the purposes of this discussion ignore that EVA 'r' also maps to the digit 2 here -- that has to do with how Brunbaugh sees variant forms that get grouped together as 'r' in EVA...)
Is there someone out there who knows how to compute the unicity distance for a cipher like this, and if so could you walk me through it?
Thanks,
Karl -
RE: Question about unicity distance
dashstofsk > 3 hours ago
This is a one-way encryption. Almost impossible to get back to the original. But why would the authors of the VMS want to do this? They themselves would not be able to decypher their own writing. And to do this for ~36000 words, and for each word to have to stop to think about which conversion to use?
It is my belief that people are thinking too hard about the manuscript. The authors probably wanted to use a more primitive method. The solution is probably more simple. -
RE: Question about unicity distance
kckluge > 3 hours ago
(3 hours ago)dashstofsk Wrote: You are not allowed to view links. Register or Login to view.This is a one-way encryption. Almost impossible to get back to the original. But why would the authors of the VMS want to do this?
The question wasn't whether Brumbaugh was wrong -- it's clear he was -- it was a specific technical question about how to compute unicity distance for such a cipher. Even with multiple options for individual words, there almost certainly has to be some text length such that only one set of possible options for the words forms a coherent text. -
RE: Question about unicity distance
oshfdk > 2 hours ago
I think this particular scheme is practically reversible, with occasional ambiguities, but only if the second stage is reversible (the digit is unambiguously identified by the glyph). Maybe with some practice it's possible to learn to read it slowly without external tools.
For example (suppose we know that the plaintext is in English and the spacing is preserved):
4631524 3933 46315 3856852 768 3925 96 62 566
Let's start with the short words: 96 and 62.
96 is (I | US | Z) + (F | O | T), given English it's either IT or IF.
62 is (F | O | T) + (B | K | R), there should be a vowel, so it starts with O and the word is OR. (Edit: or could be OK, which would work for IF OK, so it's more ambiguous than I thought, but I still think it's possible to deduce the correct plaintext.)
Now, IF OR looks odd, so it's more likely IT OR.
Then 566 and 768.
566 is (E | N | X) + (F | O | T) + (F | O | T), even without the context given by IT OR ..., I see only one word here: NOT (which means all standalone 566's in the text will be not's)
768 is (G | P | Y) + (F | O | T) + (H | Q | U), this one is very easy, given the first and the last letters are relatively rare. I can only see YOU here and no other alternatives.
So, we have:
YOU 3925 IT OR NOT. It's possible to just guess and check, but let's try and do it the slow way:
3925 is (C | L | W) + (I | US | Z) + (B | K | R) + (E | N | X), let's try 9 as I as the most frequent, especially given the first letter is a consonant (also there is a possibility of CUS/LUS too): YOU ?I?? IT OR NOT. The ending is I + (B | K | R) + (E | N | X), which given normal letter combinations in English is most likely I + (B | K | R) + E, so YOU ?I?E IT OR NOT. Not that many words would work here (C | L | W) + I + (B | K | R) + E, like WIRE, but this is most likely LIKE.
4631524 3933 46315 3856852 YOU LIKE IT OR NOT
Now we can just check if WHETHER fits 3856852 and then there are only 3 words left to crack.
3933 is (C | L | W) + (I | US | Z) + (C | L | W) + (C | L | W), etc.
I think after a while one will get used to common combinations of symbols and the decoding will get much easier. -
RE: Question about unicity distance
oshfdk > 1 hour ago
(3 hours ago)kckluge Wrote: You are not allowed to view links. Register or Login to view.The question wasn't whether Brumbaugh was wrong -- it's clear he was -- it was a specific technical question about how to compute unicity distance for such a cipher. Even with multiple options for individual words, there almost certainly has to be some text length such that only one set of possible options for the words forms a coherent text.
Given there is no answer so far, I'd say that one can just naively apply the Shannon formula, as shown in the Wikipedia article. That's what I would usually do when I have no idea what I'm doing. So,
U = Hk/D
Here Hk is the key entropy. Given no specific key constraints, this is the binary logarithm of the number of ways to group Latin alphabet (+ one extra sequence like "us" to make it split evenly) into sets of three. D is the information density in bits per character. English, as per the same article, has about 1.5 bits per character, but we reduce the source set by the factor of 3 by merging letters into groups. Without much thinking (I'm not sure I can get to the exact solution here is reasonable time
), it's likely this should increase the unicity distance, so let's just divide by 3 and say we have 0.5 bits per character.
Hk for splitting 27 items into 9 labelled groups of 3 items is (I delegated this to Claude, so won't vouch for this): lb( (27! / 3!) ** 9 ), which is roughly 70.
U = 70 / 0.5 = 140
So, you need about 140 characters of the ciphertext to identify the key for an English plaintext using this encoding scheme. I'd say this looks reasonable. -
RE: Question about unicity distance
Rafal > 29 minutes ago
Quote:So, you need about 140 characters of the ciphertext to identify the key for an English plaintext using this encoding scheme.
I'm not familiar with that measure but this link You are not allowed to view links. Register or Login to view. says:
Note that just because you have 30 characters of ciphertext for a substitution cipher, this does not mean you will be able to break it. The Unicity distance is a theoretical minimum. In general you may need 4 times more characters (100 or more) to reliably break substitution ciphers. The same problem exists with the other ciphers.
I also have a feeling that this value depends on the type of the cipher used. As we don't know the type of cipher used in VM, then we don't know how to count it. We don't even know the language which is probably a problem as well.
But Voynich Manuscript is really long. Common sense tells us that with this amount of text it should be easily broken. Yet it isn't.