Character entropy of Voynichese
#59
....While my plans to check Bennett's calculations are, as we Russians say, put off to the long box (~ dallied off), I decided to explore a simple example of a cipher of my own invention which decreases character entropy, as an illustration.

For plain text, I took the beginning of my entropy tutorial (draft). To keep the matters simple, I excluded numeration of sections, converted the rest of the numbers to words, excluded punctuation and hyphens. Thus I am left with 26 English letters, plus spaces and apostrophes - 28 characters in total. Also, there are no line breaks for simplicity. The upper/lower case is there, but it is not taken into account. 

The raw text is here:

You are not allowed to view links. Register or Login to view.
For the 1st order entropy calculation I used the following online calculator: You are not allowed to view links. Register or Login to view. (check "ignore case", uncheck "ignore space")

We see that for the plain text the figure is 4.07.

Now on to encryption. First of all, spaces are left as they are for simplicity. Now let's take a radical step towards lowering entropy. Each letter is encoded as a number, beginning with 1, so A = 1, Z = 26, and apostrophe = 27. Since we may have, say "BA" as well as "U" in the plain text, we need some means to distinguish between "21" and "21" in the encrypted text. In the former case it is the sequence of numbers 2 and 1, while in the latter case it is a single number 21. So, for teens and beyond, a dot is introduced in between to mark that the pair comprises a single number, and not a sequence of two numbers. Thus, "BA" would be "21", while "U" would be "2.1". Note that we don't need the dot in "10" and "20", since there are no standalone zeros - if a zero is encountered, it is only as element of "ten" or "twenty". Since we don't have anything like thirty and beyond, it is easy to distinguish that, say, "53" is simply "EC".

The result of the encryption is as follows:

You are not allowed to view links. Register or Login to view.
The 1st order character entropy is 3.18 which is way lower than even Voynich.

But this looks suspicious and betrays a cipher. We now need to instill this with a look of an unknown language. I would not invent a new alphabet, but just encode the numbers with pairs of letters using the table chart where A, O, I, D, S are rows, and R, N, L, M, Y, G are columns , and our numbers from 1 to 27 fill this table sequentially from left to right and from top to bottom. For example, 1 = AR, because A is the first row and R is the first column. Likewise, 2.3 = DY because number 23 happens to fit the cell of fourth row, fifth column. (I'm lazy to type the table, but surely you understand the principle). The choice of the number of rows/columns, as well as of letters, is arbitrary - I just tried to mimic the EVA bigram patterns, but I did not dedicate more than five minutes to that, so the result in this respect is not very close. However, it is good in terms of entropy - it is 3.42 - higher than with digits and dots, but lower that plain text and (still) lower than Voynich.

The text is this:

You are not allowed to view links. Register or Login to view.
Decryption is straightforward, because the cipher blocks are fixed in length (two letters each), and the process is run sequentially from left to right.

Note that the second step (digits and dots) is not really necessary. One could fill the encoding table with plain text characters at once. I just used that step to exclude ambiguities when I did "search & replace" in MS Word.

One may note the peculiar "morphology" and "grammar" of the ciphertext - some characteristic repetitiveness of the inner structure and restricted nature of word endings. We may introduce some false spaces to keep words shorter.

Of course this is not Voynichese yet - you see that all words are forced to an even number of characters, there are no gallows or gallows coverage, or some other features of the Voynich, so this is just an example.

But I have a gut feeling that something like this is going on there...
[+] 2 users say Thank You to Anton for this post
Reply


Messages In This Thread
Character entropy of Voynichese - by Anton - 23-01-2016, 02:08 PM
RE: Character entropy of Voynichese - by ReneZ - 25-01-2016, 09:28 AM
RE: Character entropy of Voynichese - by stellar - 04-11-2016, 11:18 PM
RE: Character entropy of Voynichese - by Anton - 25-01-2016, 11:53 AM
RE: Character entropy of Voynichese - by Anton - 25-01-2016, 07:36 PM
RE: Character entropy of Voynichese - by ReneZ - 26-01-2016, 08:17 AM
RE: Character entropy of Voynichese - by Davidsch - 16-03-2016, 01:20 PM
RE: Character entropy of Voynichese - by Anton - 16-03-2016, 02:37 PM
RE: Character entropy of Voynichese - by Davidsch - 18-03-2016, 11:30 AM
RE: Character entropy of Voynichese - by Anton - 18-03-2016, 12:15 PM
RE: Character entropy of Voynichese - by ReneZ - 18-03-2016, 12:19 PM
RE: Character entropy of Voynichese - by Davidsch - 18-03-2016, 04:52 PM
RE: Character entropy of Voynichese - by Sam G - 19-03-2016, 05:44 PM
RE: Character entropy of Voynichese - by Torsten - 19-03-2016, 06:37 PM
RE: Character entropy of Voynichese - by Anton - 19-03-2016, 06:19 PM
RE: Character entropy of Voynichese - by Sam G - 19-03-2016, 07:05 PM
RE: Character entropy of Voynichese - by ReneZ - 20-03-2016, 12:47 AM
RE: Character entropy of Voynichese - by Torsten - 19-03-2016, 07:32 PM
RE: Character entropy of Voynichese - by Sam G - 19-03-2016, 07:36 PM
RE: Character entropy of Voynichese - by Torsten - 19-03-2016, 07:44 PM
RE: Character entropy of Voynichese - by Sam G - 19-03-2016, 08:09 PM
RE: Character entropy of Voynichese - by Torsten - 19-03-2016, 08:33 PM
RE: Character entropy of Voynichese - by Anton - 19-03-2016, 09:26 PM
RE: Character entropy of Voynichese - by Davidsch - 19-03-2016, 10:44 PM
RE: Character entropy of Voynichese - by Torsten - 20-03-2016, 12:24 AM
RE: Character entropy of Voynichese - by Davidsch - 20-03-2016, 12:26 PM
RE: Character entropy of Voynichese - by Torsten - 20-03-2016, 02:42 PM
RE: Character entropy of Voynichese - by ReneZ - 21-03-2016, 02:24 PM
RE: Character entropy of Voynichese - by Anton - 20-03-2016, 01:41 PM
RE: Character entropy of Voynichese - by Davidsch - 20-03-2016, 06:09 PM
RE: Character entropy of Voynichese - by Torsten - 21-03-2016, 12:22 PM
RE: Character entropy of Voynichese - by Anton - 02-11-2016, 09:14 PM
RE: Character entropy of Voynichese - by Sam G - 06-11-2016, 12:13 PM
RE: Character entropy of Voynichese - by -JKP- - 06-11-2016, 12:27 PM
RE: Character entropy of Voynichese - by Sam G - 06-11-2016, 06:26 PM
RE: Character entropy of Voynichese - by Anton - 03-11-2016, 01:25 PM
RE: Character entropy of Voynichese - by Davidsch - 03-11-2016, 03:26 PM
RE: Character entropy of Voynichese - by Anton - 03-11-2016, 04:02 PM
RE: Character entropy of Voynichese - by Anton - 03-11-2016, 04:27 PM
RE: Character entropy of Voynichese - by -JKP- - 04-11-2016, 01:17 PM
RE: Character entropy of Voynichese - by Koen G - 03-11-2016, 04:31 PM
RE: Character entropy of Voynichese - by Anton - 03-11-2016, 04:34 PM
RE: Character entropy of Voynichese - by Koen G - 03-11-2016, 04:37 PM
RE: Character entropy of Voynichese - by Anton - 03-11-2016, 05:21 PM
RE: Character entropy of Voynichese - by Davidsch - 03-11-2016, 05:23 PM
RE: Character entropy of Voynichese - by Anton - 03-11-2016, 05:45 PM
RE: Character entropy of Voynichese - by Anton - 05-11-2016, 10:54 PM
RE: Character entropy of Voynichese - by Anton - 06-11-2016, 12:50 PM
RE: Character entropy of Voynichese - by -JKP- - 07-11-2016, 02:07 AM
RE: Character entropy of Voynichese - by Koen G - 07-11-2016, 05:00 AM
RE: Character entropy of Voynichese - by Anton - 07-11-2016, 02:44 PM
RE: Character entropy of Voynichese - by Anton - 10-11-2016, 06:32 PM
RE: Character entropy of Voynichese - by Anton - 05-01-2017, 12:12 AM
RE: Character entropy of Voynichese - by Koen G - 05-01-2017, 12:44 AM
RE: Character entropy of Voynichese - by Anton - 05-01-2017, 01:15 AM
RE: Character entropy of Voynichese - by Davidsch - 20-02-2017, 04:48 PM
RE: Character entropy of Voynichese - by Koen G - 20-02-2017, 05:20 PM
RE: Character entropy of Voynichese - by Davidsch - 20-02-2017, 05:22 PM
RE: Character entropy of Voynichese - by Anton - 13-12-2017, 01:25 AM
RE: Character entropy of Voynichese - by Anton - 13-12-2017, 01:47 PM
RE: Character entropy of Voynichese - by Koen G - 13-12-2017, 03:22 PM
RE: Character entropy of Voynichese - by Anton - 13-12-2017, 03:39 PM
RE: Character entropy of Voynichese - by ReneZ - 14-12-2017, 06:21 AM
RE: Character entropy of Voynichese - by ReneZ - 14-12-2017, 09:50 AM
RE: Character entropy of Voynichese - by Torsten - 14-12-2017, 04:47 PM
RE: Character entropy of Voynichese - by Anton - 14-12-2017, 01:54 PM
RE: Character entropy of Voynichese - by Anton - 14-12-2017, 04:14 PM
RE: Character entropy of Voynichese - by ReneZ - 14-12-2017, 07:02 PM
RE: Character entropy of Voynichese - by Torsten - 16-12-2017, 01:18 AM
RE: Character entropy of Voynichese - by Anton - 14-12-2017, 07:43 PM
RE: Character entropy of Voynichese - by Anton - 16-12-2017, 01:35 AM
RE: Character entropy of Voynichese - by Torsten - 16-12-2017, 10:32 PM
RE: Character entropy of Voynichese - by Anton - 16-12-2017, 10:39 PM
RE: Character entropy of Voynichese - by Torsten - 16-12-2017, 11:39 PM
RE: Character entropy of Voynichese - by -JKP- - 17-12-2017, 04:23 AM
RE: Character entropy of Voynichese - by Anton - 17-12-2017, 12:10 AM
RE: Character entropy of Voynichese - by Anton - 17-12-2017, 02:05 PM
RE: Character entropy of Voynichese - by Torsten - 17-12-2017, 02:40 PM
RE: Character entropy of Voynichese - by Anton - 17-12-2017, 02:55 PM
RE: Character entropy of Voynichese - by Torsten - 17-12-2017, 09:43 PM
RE: Character entropy of Voynichese - by Anton - 03-06-2018, 01:52 PM
RE: Character entropy of Voynichese - by Anton - 03-06-2018, 05:39 PM
RE: Character entropy of Voynichese - by Anton - 03-06-2018, 05:51 PM
RE: Character entropy of Voynichese - by -JKP- - 03-06-2018, 06:10 PM
RE: Character entropy of Voynichese - by Anton - 03-06-2018, 06:23 PM
RE: Character entropy of Voynichese - by Anton - 04-06-2018, 02:52 PM
RE: Character entropy of Voynichese - by Anton - 13-04-2019, 01:01 AM

Possibly Related Threads…
Thread Author Replies Views Last Post
  How multi-character substitution might explain the voynich’s strange entropy quimqu 23 15,116 29-06-2025, 09:37 PM
Last Post: quimqu
  How to recombine glyphs to increase character entropy? Koen G 73 63,101 11-05-2022, 07:19 PM
Last Post: Ruby Novacna

Forum Jump:


Users browsing this thread: 2 Guest(s)