Is Voynich a right-to-left script?

[nablator]

An n-gram entropy can be estimated from the corresponding n-gram frequency distribution in a text sample (whatever the tokenization).  When the text is reversed, there will be a one-to-one correspondence between n-grams in the forward and reverse directions (each "abc" forward will be represented by one "cba" in reverse, etc).

I wanted to post exactly the same question yesterday... how can there be a (global) "n-gram perplexity asymmetry"? But I did not understand the code enough to comment. Now I understand it better.

tokens = [t for t in re.findall(r"[A-Za-z0-9]+|[\u0590-\u05FF]+|[\u0600-\u06FF]+", line)]

This extracts word tokens allowing the Latin alphabet, Arabic numerals and Unicode ranges for Hebrew (\u0590-\u05FF) and Arabic (\u0600-\u06FF).

These Unicode ranges should be improved, they include punctuation:
You are not allowed to view links. Register or Login to view.
You are not allowed to view links. Register or Login to view.

Code:

def ngram_logprob(word, counts, context_counts, tokenizer, n=3):     tokens = tokenizer(word)     logp, V = 0.0, len(counts)     for i in range(len(tokens)-n+1):         ngram = tuple(tokens[i:i+n])         context = ngram[:-1]         count = counts.get(ngram, 0) + 1         context_total = context_counts.get(context, 0) + V         logp += math.log(count/context_total)     return logp

Note: tokenizer actually splits words into characters:

Code:

def tokenize_plain_word(word):     """Neutral: split into characters only, no prefix/suffix rules."""     return list(word)

If I understand correctly this sums the log of the probability of ngram in the context of ngram[:-1] (this removes the last element, thus the asymmetry) in each word. V seems useless (adds 0.0).