The Voynich Ninja
Character entropy of Voynichese - Printable Version

+- The Voynich Ninja (https://www.voynich.ninja)
+-- Forum: Voynich Research (https://www.voynich.ninja/forum-27.html)
+--- Forum: Analysis of the text (https://www.voynich.ninja/forum-41.html)
+--- Thread: Character entropy of Voynichese (/thread-148.html)

Pages: 1 2 3 4 5 6 7 8 9 10


RE: Character entropy of Voynichese - Sam G - 19-03-2016

(19-03-2016, 07:32 PM)Torsten Wrote: You are not allowed to view links. Register or Login to view.
Quote:I've responded to this point of yours before:

You are not allowed to view links. Register or Login to view.

I've responded to your point that there is some word order in the VMS (see You are not allowed to view links. Register or Login to view.):

Yeah, but you avoided responding to the part about how there might be rules affecting word order that we aren't aware of, just as you are avoiding responding to it now.


RE: Character entropy of Voynichese - Torsten - 19-03-2016

Quote:Yeah, but you avoided responding to the part about how there might be rules affecting word order that we aren't aware of, just as you are avoiding responding to it now.

No, I didn't respond because I didn't see your response. Even now I didn't found your response.

What did you mean with "rules affecting word order that we aren't aware of". The point is that word order is absent for the VMS. With other words there are no rules concerning the word order.
If you think otherwise please demonstrate some of the rules we aren't aware of.


RE: Character entropy of Voynichese - Sam G - 19-03-2016

(19-03-2016, 07:44 PM)Torsten Wrote: You are not allowed to view links. Register or Login to view.
Quote:Yeah, but you avoided responding to the part about how there might be rules affecting word order that we aren't aware of, just as you are avoiding responding to it now.

No, I didn't respond because I didn't see your response. Even now I didn't found your response.

What did you mean with "rules affecting word order that we aren't aware of". The point is that word order is absent for the VMS. With other words there are no rules concerning the word order.
If you think otherwise please demonstrate the rules we aren't aware of.

An example of a rule affecting word order from English would be the different structures used to create declarative sentences vs. questions, e.g. "you have eaten" vs. "have you eaten?"  The fact that these two sentences both exist doesn't mean that English has no word order and therefore no grammar. 

That's just one example.  Word order is used in many different ways in different languages.  Your assumption seems to be that languages have a single fixed word order that is used in all cases, and that's wrong.


RE: Character entropy of Voynichese - Torsten - 19-03-2016

Quote:Your assumption seems to be that languages have a single fixed word order that is used in all cases, and that's wrong.

Nope, my point is not that there must be a fixed word order. My point is that beside that similar words do co-occur no word order exist for the VMS. With other words, my point is that there is no structure or word order. The VMS is only containing words similar to each other. 

Moreover similar words are related to each other. Similar words occur with comparable frequency, whereas types which contain less frequent glyphs or bigrams in most cases occur less frequently. It is possible to describe a system for the VMS. But this would be a system of similar words. In natural languages a word normally (cf. poems) is used because of its meaning and not because it is similar to a previously written one. (see You are not allowed to view links. Register or Login to view. as Timm 2016 p. 7)


RE: Character entropy of Voynichese - Anton - 19-03-2016

Quote:Well, it can't be a ciphertext of any language which has a higher entropy than the VMS text (which rules out all European languages and many others)

I don't think that the VMS text has been compared with all European languages of the time (Early New High German, for example). What I saw in the Stallings' paper were just Latin and English, and the latter not earlier than 16th century version.


RE: Character entropy of Voynichese - Davidsch - 19-03-2016

I made an example (english) text in 15 minutes, which has almost the same characteristics in my opinion. 
Yes it has a little bit more letters, but i am interested if you can let me know the differences in statistical behaviour. 



example text
here you find the herbal which is used for celebration of easter find them near to other herbs in the field will grow flowers in summer has white flower and can be used in kitchen as well as for treating the wounds someone who is badly hurt on the skin can not used in salads because the kind will not allow this to be eaten by women however you still grind it and use it for bathing to do so you must first dry the flowers and throw away the rest then when flowers are dry pulverize them until they are nothing more than small particles than add some olive oil and make soap the usual way herbal is very common 

changed into (space => dot.  processed 121 words . processed 491 letters) 
hery.fov.find.thy.herbal.which.ys.vsed.fog.celebration.ef.easteg.find.them.neag.to.etheg.herbs.yn.thy.field.will.prow.flowers.yn.svmmeg.has.whity.floweg.
end.can.by.vsed.yn.kitchen.es.well.es.fog.treating.thy.wovnds.someony.who.ys.badly.hvrt.en.thy.skin.can.not.vsed.
in.salads.becavsy.thy.kind.will.not.ellow.this.to.by.eaten.by.women.howeveg.fov.still.prind.yt.end.vsy.yt.fog.bathing.to.do.so.fov.mvst.first.dry.thy.flowers.end.throw.eway.thy.rest.then.when.flowers.ery.dry.pvlverizy.them.vntil.they.ery.nothing.mory.than.small.particles.than.edd.somy.elivy.eil.end.maky.soap.thy.vsval.way.herbal.ys.very.common.


RE: Character entropy of Voynichese - Torsten - 20-03-2016

@Davidsch: Sorry, but your text doesn't share the same characteristics.


Typical for the VMS is that similar words occur with comparable frequency, whereas types which contain less frequent glyphs or bigrams in most cases occur less frequently. Starting from one of the three core words 'daiin', 'ol' and 'chedy' it is possible for the VMS to build a grid:
 ol (537)   | al (260) | dol (117) | dal (253) |  sol ( 75)  | sal ( 55)
 or (363)   | ar (350) | dor ( 73) | dar (318) |  sor ( 57)  | sar ( 84)
 om ( 22)  | am ( 88) | dom ( 7)  | dam ( 98) |  som ( 1)  | sam ( 10)
 os ( 29)   | as ( 5)  | dos ( 1)  | das ( 4) | sos ( 8)   | sas ( 2) 
(see You are not allowed to view links. Register or Login to view. as Timm 2014 p. 70)
Such core words does not exist for your text. The most frequent words in your text are 'thy' (8 times) and 'end' (4 times). But they are not similar to other common words like 'flowers' (3 times) and 'vsed' (3 times). Therefore my conclusion is that words similar to each other are not common enough.

Other features not similar to the VMS are:
Similar words like 'fov' and 'fog' do not co-occur. Even the short text contains two phrases where a word is framed by a repeated short word: 'es.well.es' and 'by.eaten.by'.
The word length distribution is not binomial (There are too many short words).


RE: Character entropy of Voynichese - ReneZ - 20-03-2016

(19-03-2016, 06:19 PM)Anton Wrote: You are not allowed to view links. Register or Login to view.[...] the "anomalously low entropy" of Voynichese is something at least not consistently proven, [...]

I'm afraid that I don't understand what you mean exactly.

It is one of the easiest features of the MS that anyone trying to explain its meaning should ideally understand.


RE: Character entropy of Voynichese - Davidsch - 20-03-2016

(20-03-2016, 12:24 AM)Torsten Wrote: You are not allowed to view links. Register or Login to view.@Davidsch: Sorry, but your text doesn't share the same characteristics.

.....

The word length distribution is not binomial (There are too many short words).

Torsten:  i was focussing on the entropy here.  Too many short words? I am sure the text is too short to show or not show binomial distribution. 

I will get back on your "core similarity modifications" in about a couple of weeks, perhaps more time is needed cause I will have to do some programming for that.
Currently i am in the middle of another task.


RE: Character entropy of Voynichese - Anton - 20-03-2016

(20-03-2016, 12:47 AM)ReneZ Wrote: You are not allowed to view links. Register or Login to view.
(19-03-2016, 06:19 PM)Anton Wrote: You are not allowed to view links. Register or Login to view.[...] the "anomalously low entropy" of Voynichese is something at least not consistently proven, [...]

I'm afraid that I don't understand what you mean exactly.

It is one of the easiest features of the MS that anyone trying to explain its meaning should ideally understand.


Characterizing something as "high" or "low", and "anomalously" in particular, implies comparison with something "commonly" "low" or "high", respectively. The procedure of comparison requires that the objects under consideration are comparable.

As I have been noting, character entropy is alphabet-dependent; next, direct comparison of entropy figures between languages with different alphabets is methodologically wrong - I guess, redundancy should be compared instead; next, comparison of the text dated no later than 15th century with texts in languages of 16th century and later is also methodologically wrong.

In these circumstances, I can not accept "low" Voynichese character entropy at face value. It might be "low" indeed, but someone should show that in a consistent way:

- checking how the entropy changes with most suggestive variations in Voynichese alphabet (e.g. considering ckh as one symbol instead of three or considering ir as one symbol instead of three, and so on);

- comparing redundancies rather than absolute entropies;

- making some comparisons (at least) to texts in medieval European languages.