What Lies Beneath: Statistical Structure in Voynichese Revealed by Transformers

[quimqu]

Interesting work. I agree with you (and many others) that the structure of Voynich word types is rather regular, with some underlying 'grammar', and I think the method you used is surely interesting.

I had my share of fun working on grammars (actual slot grammars, in my case). Now I don't have the time at hand to check your GitHub repository and see what I can understand, I confess I did not understand much of how your method works and what it actually does, but I'll try tomorrow.

One question: can you apply your method in 'reverse', I mean: use your GPT model to decompose the original Voynich word types in parts ('tokens'), such as "qo", "y", "dy", "aiin" and so on, for each word (some of which may turn out not to be conforming to the grammar)? In this case it could be possible to compare the results with different approaches (I defined a metric which ranks different grammars according to how many bits of information are needed to represent the text by using optimal encoding on the words divided in 'tokens').

Hi,
Thanks a lot for your message. I find your suggestion about grammar-based decomposition really interesting — that’s exactly the kind of direction I’d like to explore next.
To clarify what I’ve done so far:
I trained several small GPT models on Voynichese using a character-level tokenizer, based on the Currier transliteration. This worked quite well: the model was able to predict character sequences with low perplexity (~3.3), which suggests a high degree of internal structure.
In contrast, using word-level tokenization (based on dot-separated EVA words) gave very poor results — mainly because of the large vocabulary size and lack of training data per token.

---

At this point, I’m considering two directions:

1. Trying more modern transliterations (like Takahashi or updated EVA versions). But I’m a bit concerned that these are too detailed — they distinguish rare glyphs very precisely, which might make it harder for the model to generalize.
   
2. Switching to syllable-like units instead of characters — which is exactly what you suggested.
   I’d love to hear your opinion on this:
   

• What kind of syllables (e.g. "qo", "dy", "aiin", etc.) do you think would make sense?
  
• Which transliteration would be the best basis for such tokenization?
  

If you’ve already explored this type of decomposition, I’d be really interested in hearing more or comparing approaches.
Thanks again for your input!