NOTE!!!! This is PURELY ME TESTING, trying to get stable, reliable output from GPT.
Nothing in this post is claimed to be FACTUAL, ACCURATE or even SANE!
Nothing in this post is claimed to be FACTUAL, ACCURATE or even SANE!
After getting the new and hopefully better parser running today I did some more testing. I've been curious about language structure, which I'm still learning, and allowed GPT a bit of freedom (still locked into my parser) in order to explore vowels and consonants and their structure.
It came up with an algorithm with bits borrowed from Shannon, Zipf, and Kim & Snyder. Called the behavior model, its derived from information theory and Kim & Snyder's AI work hunting for vowels in hundreds of languages, but in a simpler, form. It works by watching how each letter behaves in words to see which ones move around easily like vowels and which ones stay stuck like consonants. It also created a combinatorial method which works by seeing which letters can fit in lots of different places and mix with many other letters, like vowels do. Then, I had it pull known algorithms for checking vowels and consonants and apply their rules to the Voynich.
It grabbed the Sukhotin algorithm which creates the vowel set everyone seems to agree on (a, o, e, i, y). It grabbed Jacques Guy Neighbor's - diversity algorithm where each letter is scored by the number of distinct preceding and following letters. It grabbed the method used by Kim & Snyder where they trained an AI to find vowels and consonants across hundreds of languages. Because I don't have access to their training or data, it modified it to read the Voynich in the same way. And it grabbed a method Kevin Knight used in his work on deciphering cryptographs. It labeled the Kim & Snyder adaptation as combinitorial and the Knight adaptation as stability.
I had it run all of these tests against English (Dracula from Gutenberg). If it had it right, they should mostly be able to detect the English vowel set (a, e, i, o, u). Here's the results of that test.
I then had it run it's behavior model against the Voynich. It came up with these possible vowels.
(o, a, d, f, p, t, k)
I specifically quizzed it about the missing accepted vowels (e, i, y). It discounted all 3 because they didn't have enough occurrences where they began or ended a word. Most of them, were in the middle of a word, by a significant percentage.
So, using it's 'found' vowels as a starting point, I had it compare what the other methods found and see if they agreed with it. Here's the result.
Finally, I had it run all the methods and take all of their output and compare them all side by side. Here's the result of that test.
So, all of them agreed a and o are vowels. Most agreed on d and k and from there, they were split 50/50 or less on what's a vowel and what isn't.
And for those who can run python, I had it create a .py file of these methods which you are free to test, rip apart or ignore. The parser .py file it refers to can be found in my previous post. The pagesH.txt is Takahashi's transcription as downloaded directly from his site.
Thoughts: Seeing it pull the expected vowels from Dracula reasonably accurately and seeing the Sukhotin method pull the expected 'default' Voynich vowels from the parsed output gave me some extra confidence that the parser worked as intended. Can the results be trusted? Hell no. But, if confirmed, it's vowel detection method produced some interesting results.
Code:
import re
import math
import random
from collections import Counter, defaultdict
# =========================
# 0. CORPUS HELPERS
# =========================
def load_english_words(path):
"""
Load an English corpus (e.g., Dracula from Project Gutenberg).
- Strips Gutenberg header/footer using *** START/END OF marker.
- Lowercases.
- Keeps only a–z as letters; all else → space.
- Returns: list of word strings.
"""
words = []
in_text = False
with open(path, "r", encoding="utf-8") as f:
for line in f:
if "*** START OF" in line:
in_text = True
continue
if "*** END OF" in line:
break
if not in_text:
continue
line = line.lower()
line = re.sub(r"[^a-z]+", " ", line)
parts = line.split()
words.extend(p for p in parts if p)
return words
def load_voynich_words(pages_path, parser_module):
"""
Load Voynich words using the locked Takahashi parser module.
`parser_module` is an imported module providing:
- get_folio_list(pages_path)
- parse_folio_corpus(pages_path, folio_id)
Returns: list of word strings (already in Takahashi/EVA charset).
"""
words = []
for fid in parser_module.get_folio_list(pages_path):
text = parser_module.parse_folio_corpus(pages_path, fid)
if not text:
continue
for w in text.split():
if w:
words.append(w)
return words
# =========================
# 1. CORE STATISTICS
# =========================
def build_stats(words):
"""
From a list of words, compute:
- alphabet: sorted list of distinct glyphs
- counts[g]: total count
- pos[g]['initial'|'medial'|'final']
- prev[g], nxt[g]: neighbor frequency Counters
Returns: (alphabet, counts, pos, prev, nxt)
"""
counts = Counter()
pos = defaultdict(lambda: Counter())
prev = defaultdict(Counter)
nxt = defaultdict(Counter)
for w in words:
if not w:
continue
chars = list(w)
L = len(chars)
for i, ch in enumerate(chars):
counts[ch] += 1
if i == 0:
pos[ch]["initial"] += 1
elif i == L - 1:
pos[ch]["final"] += 1
else:
pos[ch]["medial"] += 1
if i > 0:
prev[ch][chars[i - 1]] += 1
if i < L - 1:
nxt[ch][chars[i + 1]] += 1
alphabet = sorted(counts.keys())
return alphabet, counts, pos, prev, nxt
def entropy(counter, base=2):
"""
Normalized entropy of a frequency Counter.
Returns 0.0 if empty.
"""
total = sum(counter.values())
if total == 0:
return 0.0
e = 0.0
for v in counter.values():
if v > 0:
p = v / total
e -= p * math.log(p, base)
# normalize by max entropy for this support size, using len(counter)
if len(counter) <= 1:
return 0.0
max_e = math.log(len(counter), base)
return e / max_e if max_e > 0 else 0.0
def positional_entropy(counts, pos_counts):
"""
Compute positional entropy for each glyph over {initial, medial, final}
using log base 3, already normalized (since max is log_3(3) = 1).
Returns: dict[glyph] -> entropy in [0,1].
"""
H = {}
for g, total in counts.items():
if total == 0:
H[g] = 0.0
continue
init = pos_counts[g]["initial"] / total
med = pos_counts[g]["medial"] / total
fin = pos_counts[g]["final"] / total
probs = [init, med, fin]
e = 0.0
for p in probs:
if p > 0:
e -= p * math.log(p, 3)
H[g] = e # already normalized
return H
# =========================
# 2. SUKHOTIN ALGORITHM
# =========================
def sukhotin_vowels(words):
"""
Sukhotin's classic adjacency-based vowel detection algorithm.
Returns: set of glyphs classified as vowels.
"""
alphabet, _, _, _, _ = build_stats(words)
# symmetric adjacency matrix F
F = {a: Counter() for a in alphabet}
for w in words:
if len(w) < 2:
continue
for i in range(len(w) - 1):
x, y = w[i], w[i + 1]
if x in F and y in F:
F[x][y] += 1
F[y][x] += 1
# initial r[i]
r = {a: sum(F[a].values()) for a in alphabet}
vowels = set()
while True:
m = max(r, key=r.get)
if r[m] <= 0:
break
vowels.add(m)
# update neighbors
for i in alphabet:
if F[i][m] > 0:
r[i] -= 2 * F[i][m]
F[i][m] = 0
F[m][i] = 0
r[m] = 0
return vowels
# =========================
# 3. JACQUES GUY NEIGHBOR DIVERSITY
# =========================
def guy_scores(words):
"""
Jacques Guy-style neighbor diversity score:
score(g) = #distinct previous neighbors + #distinct next neighbors
Returns: dict[glyph] -> score.
"""
_, _, _, prev, nxt = build_stats(words)
scores = {}
for g in sorted(prev.keys() | nxt.keys()):
scores[g] = len(prev[g]) + len(nxt[g])
return scores
# =========================
# 4. BEHAVIORAL MODEL (OUR METHOD)
# =========================
def behavioral_scores(words):
"""
Our composite vowel-likeness score per glyph:
behavioral(g) =
+ medial_ratio
+ positional_entropy
+ 0.5 * (prev_entropy + next_entropy)
- boundary_ratio
Where:
medial_ratio = medial / total
boundary_ratio = (initial + final) / total
positional_entropy: over {initial, medial, final}, base 3
prev/next_entropy: normalized neighbor entropies, base 2
Returns: dict[glyph] -> score (higher = more vowel-like).
"""
alphabet, counts, pos, prev, nxt = build_stats(words)
pos_H = positional_entropy(counts, pos)
scores = {}
for g in alphabet:
total = counts[g]
if total == 0:
scores[g] = 0.0
continue
init = pos[g]["initial"]
med = pos[g]["medial"]
fin = pos[g]["final"]
medial_ratio = med / total
boundary_ratio = (init + fin) / total
H_pos = pos_H[g]
H_prev = entropy(prev[g], base=2)
H_next = entropy(nxt[g], base=2)
score = medial_ratio + H_pos + 0.5 * (H_prev + H_next) - boundary_ratio
scores[g] = score
return scores
# =========================
# 5. COMBINATORIAL METHOD (KNIGHT-INSPIRED)
# =========================
def combinatorial_scores(words):
"""
Knight-inspired combinatorial scoring:
score(g) =
+ 0.5 * (|Prev(g)| + |Next(g)|)
+ 5 * avg_neighbor_entropy
+ 10 * positional_entropy
- |Prev/Next|: neighbor diversity
- avg_neighbor_entropy: mean of normalized prev/next entropies
- positional_entropy: as above
Constants (5, 10) are fixed scaling factors, applied identically across
languages and corpora, chosen to keep terms numerically comparable.
Returns: dict[glyph] -> score (higher = more nucleus-like).
"""
alphabet, counts, pos, prev, nxt = build_stats(words)
pos_H = positional_entropy(counts, pos)
comb = {}
for g in alphabet:
total = counts[g]
if total == 0:
comb[g] = 0.0
continue
uniq_prev = len(prev[g])
uniq_next = len(nxt[g])
H_prev = entropy(prev[g], base=2)
H_next = entropy(nxt[g], base=2)
H_neighbors_avg = 0.5 * (H_prev + H_next)
H_pos = pos_H[g]
score = 0.5 * (uniq_prev + uniq_next) + 5 * H_neighbors_avg + 10 * H_pos
comb[g] = score
return comb
# =========================
# 6. KIM & SNYDER–STYLE CLUSTERING (ADAPTED)
# =========================
def bayesian_style_clusters(words, seed=42):
"""
Unsupervised 2-means clustering over behavioral feature vectors.
This is an adaptation in the spirit of Kim & Snyder (2013).
Features per glyph g:
[ behavioral_score(g) ]
We then:
- run 2-means on this 1D feature,
- choose cluster with higher mean behavioral score as vowel-like.
Returns:
vowel_like: list of glyphs in vowel-ish cluster
consonant_like: list of glyphs in other cluster
"""
beh = behavioral_scores(words)
glyphs = sorted(beh.keys())
data = [beh[g] for g in glyphs]
if len(glyphs) < 2:
return glyphs, []
random.seed(seed)
# initialize centroids using min and max behavior scores
c0 = min(data)
c1 = max(data)
centroids = [c0, c1]
for _ in range(50):
clusters = {0: [], 1: []}
for i, v in enumerate(data):
d0 = (v - centroids[0]) ** 2
d1 = (v - centroids[1]) ** 2
label = 0 if d0 <= d1 else 1
clusters[label].append(i)
new_centroids = []
changed = False
for k in (0, 1):
if not clusters[k]:
new_centroids.append(centroids[k])
continue
mean = sum(data[i] for i in clusters[k]) / len(clusters[k])
if abs(mean - centroids[k]) > 1e-9:
changed = True
new_centroids.append(mean)
centroids = new_centroids
if not changed:
break
# pick vowel cluster = higher average behavioral score
def cluster_mean(k):
idxs = clusters[k]
if not idxs:
return float("-inf")
return sum(data[i] for i in idxs) / len(idxs)
vowel_cluster = max((0, 1), key=cluster_mean)
vowel_like = [glyphs[i] for i in clusters[vowel_cluster]]
consonant_like = [glyphs[i] for i in clusters[1 - vowel_cluster]]
return vowel_like, consonant_like
# =========================
# 7. CROSS-SECTIONAL STABILITY
# =========================
def stability_by_chunks(words, vowels, n_chunks=8):
"""
Split corpus into n_chunks and compute vowel distribution stability.
For each chunk:
- count occurrences of each candidate vowel in that chunk
- compute fraction of total vowel tokens due to each candidate
Returns: list of dicts:
[ {vowel: proportion_in_chunk, ...}, ... ]
"""
if not words:
return []
all_text = "".join(words)
chunk_len = max(1, len(all_text) // n_chunks)
results = []
for k in range(n_chunks):
start = k * chunk_len
end = (k + 1) * chunk_len if k < n_chunks - 1 else len(all_text)
sub = all_text[start:end]
c = Counter(ch for ch in sub if ch in vowels)
total = sum(c.values())
if total == 0:
results.append({v: 0.0 for v in vowels})
else:
results.append({v: c[v] / total for v in vowels})
return results
# =========================
# 8. SIMPLE DRIVER / EXAMPLE
# =========================
if __name__ == "__main__":
# Example usage outline (user must adjust paths):
# English example:
# eng_words = load_english_words("345-0.txt")
# print("Sukhotin (EN):", sukhotin_vowels(eng_words))
# print("Behavioral top (EN):", sorted(behavioral_scores(eng_words).items(),
# key=lambda x: x[1], reverse=True)[:10])
# Voynich example (requires locked parser module):
# import takahashi_parser_locked_export_API_GUARDED as vp
# voy_words = load_voynich_words("PagesH.txt", vp)
# print("Sukhotin (VMS):", sukhotin_vowels(voy_words))
# print("Behavioral top (VMS):", sorted(behavioral_scores(voy_words).items(),
# key=lambda x: x[1], reverse=True)[:10])
pass