[Article] Entropy Analysis of Questionable Text Sources by Example of the Voynich Manuscript
#4
The paper is deriving its conclusions entirely from the frequency distribution of single characters.

The first comparison seems to be a visual comparison of the shape of this distribution.
The other two are the regular single-character entropy as it has been used here already (called the Shannon entropy), and the more general Renyi entropy, also applied to single characters.

I had not heard of this before, but what the paper says, combined with a quick look at Wikipedia, clarify what this is about. This is an entropy with a variable parameter alpha, which may run from 0 to infinity.
If alpha = 0 one gets what we have called the H0. If alpha = 1 one gets the Shannon entropy (H1). If alpha goes to infinity, one gets the negative logarithm of the probability of the most frequent character.

Therefore, if two character distributions are similar, also the Renyi entropy as a function of alpha will be similar.

This is what happens in the comparison with plain texts, and this is what the paper shows.

What remains mysterious is that the number of different characters in the various plain texts are extremely large, e.g. 140 for an Italian text. Even counting upper and lower case, the various accented characters, and interpunction, I can't understand where that comes from.
[+] 2 users say Thank You to ReneZ for this post
Reply


Messages In This Thread
RE: Entropy Analysis of Questionable Text Sources by Example of the Voynich Manuscript - by ReneZ - 03-10-2019, 08:18 AM

Possibly Related Threads…
Thread Author Replies Views Last Post
  [Article] Analysis of the relation between words within the Voynich Manuscript Torsten 2 9,669 09-11-2022, 09:40 AM
Last Post: nablator
  [Article] Orlov YN - Language recognition methods and Voynich Manuscript analysis Scarecrow 7 11,459 26-08-2022, 12:08 AM
Last Post: Anton
  [Article] "Principal component analysis of characters in the Voynich manuscript Torsten 9 14,979 04-03-2020, 07:39 AM
Last Post: Alin_J
  [Article] A Hidden Markov Model for the Linguistic Analysis of the Voynich Manuscript Torsten 8 12,770 22-10-2019, 05:56 PM
Last Post: MarcoP
  [Article] Statistical regularity of European languages and Voynich Manuscript analysis Anton 15 22,275 22-10-2019, 01:07 PM
Last Post: Monica Yokubinas
  [Article] Unconventional Methods in Voynich Manuscript Analysis Torsten 25 27,637 08-08-2019, 08:08 PM
Last Post: Koen G

Forum Jump:


Users browsing this thread: 2 Guest(s)