Is Voynich a right-to-left script?

[obelus]

If "perplexity" is in vogue for language models, fine... an earlier thread explained that perplexity is monotonically related to the Shannon entropy.  If so, these directionality results are puzzling.

An n-gram entropy can be estimated from the corresponding n-gram frequency distribution in a text sample (whatever the tokenization).  When the text is reversed, there will be a one-to-one correspondence between n-grams in the forward and reverse directions (each "abc" forward will be represented by one "cba" in reverse, etc).  Therefore any function that can be calculated from n-gram frequencies alone, such as "absolute" entropy, or conditional entropy to any order, must be the same in either direction.  How does perplexity break this symmetry?

If the text samples were not globally reversed, character by character as claimed, then statistical asymmetry may be have been introduced when correlations at the word or line level were altered.

[nablator]

An n-gram entropy can be estimated from the corresponding n-gram frequency distribution in a text sample (whatever the tokenization).  When the text is reversed, there will be a one-to-one correspondence between n-grams in the forward and reverse directions (each "abc" forward will be represented by one "cba" in reverse, etc).

I wanted to post exactly the same question yesterday... how can there be a (global) "n-gram perplexity asymmetry"? But I did not understand the code enough to comment. Now I understand it better.

tokens = [t for t in re.findall(r"[A-Za-z0-9]+|[\u0590-\u05FF]+|[\u0600-\u06FF]+", line)]

This extracts word tokens allowing the Latin alphabet, Arabic numerals and Unicode ranges for Hebrew (\u0590-\u05FF) and Arabic (\u0600-\u06FF).

These Unicode ranges should be improved, they include punctuation:
You are not allowed to view links. Register or Login to view.
You are not allowed to view links. Register or Login to view.

Code:

def ngram_logprob(word, counts, context_counts, tokenizer, n=3):     tokens = tokenizer(word)     logp, V = 0.0, len(counts)     for i in range(len(tokens)-n+1):         ngram = tuple(tokens[i:i+n])         context = ngram[:-1]         count = counts.get(ngram, 0) + 1         context_total = context_counts.get(context, 0) + V         logp += math.log(count/context_total)     return logp

Note: tokenizer actually splits words into characters:

Code:

def tokenize_plain_word(word):     """Neutral: split into characters only, no prefix/suffix rules."""     return list(word)

If I understand correctly this sums the log of the probability of ngram in the context of ngram[:-1] (this removes the last element, thus the asymmetry) in each word. V seems useless (adds 0.0).

[nablator]

John Winstead's method reveals statistical differences between the start and the end of words:

I imagine this method only works because the tested languages use suffixes more than prefixes for You are not allowed to view links. Register or Login to view..

There may be some (?) exceptions:

In head-marking languages, the adpositions can carry the inflection in adpositional phrases. This means that these languages will have inflected adpositions.

[quimqu]

In the case of the Voynich, I think having a lower perplexity in the RTL direction is not really meaningful. Perplexity, which is directly related to entropy, measures how surprised the model is when it sees the actual next character (or word). We already know that the Voynich has unusually low entropy for bigrams compared to natural languages. So, in my opinion, a slightly lower entropy or perplexity in the RTL direction doesn’t add much insight. In fact, it might even be harder to explain why the text would have an even lower entropy than it already does.

[RobGea]

Nevermind

[magnesium]

In the case of the Voynich, I think having a lower perplexity in the RTL direction is not really meaningful. Perplexity, which is directly related to entropy, measures how surprised the model is when it sees the actual next character (or word). We already know that the Voynich has unusually low entropy for bigrams compared to natural languages. So, in my opinion, a slightly lower entropy or perplexity in the RTL direction doesn’t add much insight. In fact, it might even be harder to explain why the text would have an even lower entropy than it already does.

One potentially useful test would be to assess the perplexity of a Naibbe ciphertext in LTR and RTL directions. In this case, the cipher is very consciously mimicking the VMS word grammar (the driver of the VMS’s entropy) while encrypting a left-to-right Latin or Italian plaintext.

[Labyrinthinesecurity]

I have updated the notebook to calculate the Gini index, and reproduced John W’s findings on english (positive delta Gini) and Hebrew (negative)
For Voynich it is unconclusive (delta zero), meaning Voynich fails the Ashraf &Sinha hypothesis, unlike any other tested language
So perplexity seems a better way of measuring directionality, for Voynich at least

Why have you chosen perplexity as your only statistical measure?
How would perplexity be expected to reveal information about the directionality of the script ?
What happens if you reverse the text in your corpora i.e "aroproc ruoy ni txet eht esrever uoy fi sneppah tahW" ?

[luiscrassus]

The Voynich manuskript is written from right to left! I wrote this ten years ago.

[Jorge_Stolfi]

The Voynich manuskript is written from right to left! I wrote this ten years ago.

On almost every line and label there are clues that it was written left to right. What is the evidence to the contrary?

[luiscrassus]

Its´s the uneven spacing between the words and the stepped lines. Try writing on a blank piece of paper from right to left and you´ll get the same steps. Nobody wants to see that.