Measuring Long-Range Structure in the Voynich Manuscript

[quimqu]

It is this formula. It is essentially the mutual information between characters separated by exactly d positions, averaged over all such pairs in the corpus.

That's what worries me (potentially, I don't know, but I suspect a possible issue there): how do you average the MI of all the pairs at distance d? A simple mean?

For each distance d, I do not average individual MI values. I treat all character pairs at that distance as samples of two random variables: the character at position t and the character at position t+d. I then compute the mutual information between those two variables in the standard way, using their empirical joint and marginal distributions. So there is only one MI(d) per distance, not an average of many MIs. This is the python code for the calculation:

Code:

# ========================= # MI(d) estimator (sparse joint, smoothed marginals) # ========================= def mutual_information_lag_sparse(ids, lag, alpha=0.5):     x = ids[:-lag]     y = ids[lag:]     n = len(x)     if n <= 20:         return np.nan     V = int(max(ids)) + 1     cx = np.bincount(x, minlength=V).astype(np.float64)     cy = np.bincount(y, minlength=V).astype(np.float64)     px = (cx + alpha) / (n + alpha * V)     py = (cy + alpha) / (n + alpha * V)     joint = defaultdict(int)     for a, b in zip(x, y):         joint[(int(a), int(b))] += 1     mi = 0.0     inv_n = 1.0 / n     for (a, b), cnt in joint.items():         pxy = cnt * inv_n         mi += pxy * (math.log(pxy) - math.log(px[a]) - math.log(py[b]))     return float(mi)