12-12-2025, 01:51 PM
Hey folks,
I've been binging Koen's videos on Youtube over the last week, great stuff!
Obviously that means I'm a Voynich novice, but I did use computational linguistics during my PhD at the MPI in Nijmegen, Netherlands, so I couldn't help but dig my fingers into the data :)
I'm not claiming any novelty but I haven't seen the different analysis steps put together in one place so I figured I might as well publish it here. (However, I do think in the end, I have some interesting results that I didn't see anywhere else... but more about this below and in the next post.)
I put the data, scripts and a small analysis report on a dedicated Github repository, if somebody wants to have a deeper look: You are not allowed to view links. Register or Login to view.
My main idea for this round is to perform an TF-IDF analysis. This is a statistical tool where you count how often words occur over the whole text, vs the individual text segments (pages). With this method, one can (approximately) distinguish "content words" from "function words". Content words are those that are specific to a particular topic, like "Voynich" or "Quantum", while function words are words that show up everywhere, like "the", "of", "and", etc. I think it would be tantalizing to produce a list of Voynich words, where we can guess, from section and illustration cues, what they might mean, given where they show up. (Although I don't think it would bring us closer to deciphering the text, it would be fun.)
Down the line, maybe I have time to produce a visual tool where people can explore how words cluster in certain portions of the text. Not quite as fancy as the amazing tool on voynichese.com but in the same spirit.
I'm currently getting into working with LLMs (building them, not talking to Chat GPT) and I am very curious if one can use these tools to identify semantic clusters of Voynich words. Tbd.
I obviously haven't read everything there is on Voynich, but I did my best to go through voynich.nu and Bowern and Lindemann (2020) as well as the latest and pinned posts on this forum, to get a base understanding what's commonly known and what's currently under discussion. I'm looking forward to learn more from you.
I want to start by stating my base preconceptions/assumptions when I went into the analysis, as well as some questions that maybe you can help me with.
Assumptions:
1. The text is real in the sense that somebody in the 15th century wrote something down to communicate information to somebody else.
2. The transcription is reasonably good and conveys the textual content of the VMS to an overwhelming degree, so we can base an analysis on it.
3. The words are words in the sense that they can, through translation, combination, compression, augmentation by auxiliary information, or some other process, be rendered into a language that someone at some time spoke. If there is a cipher, it did not jumble words by moving word boundaries or similar shenanigans.
4. Letters are only meaningful with regards to the manuscript itself. They cannot be identified in a one-to-one manner with any language.
5. The manuscript was written by several scribes/authors, possibly at different times, possibly without knowing each other. The known separations are Currier A and B as well as the 5 Hands (Davis, Lisa Fagin. 2020).
6. There is no hope of me ever decrypting the text by myself since I have none of the necessary skills to actually understand any language that the authors spoke, even less the manuscript itself.
Questions:
1. The whole analysis is based on IT2a-n.txt from You are not allowed to view links. Register or Login to view. Is this the correct choice? As far as I understand, it's a version of the TT transcription, but I don't know what the state of the art is. I noticed that the transcription used on voynichese.com is different and in some places more complete, but I don't know.
2. How is the interaction between "fan groups" like this one and the academic community? From what I saw in the videos, there is a fairly good collaboration, but I still wonder. I know that for some topics that garner so much public interest, there can be a lot of tension.
3. I had trouble finding a "definitive" distinction of pages into Currier A and B. I don't know if that's because it is not fully defined, if there is disagreement or if I just didn't look at the right places.
Base results:
Before we get to the good stuff, I want to post the base analysis as a sanity check but also so they are all in one spot. As I said, these are all well known but it's good for me to see the data myself, so maybe also for others.
I split up all the analysis steps by Currier A and B. Going in, I did not have any idea how close both languages are. My initial assumption was actually that they are as different as German and Latin. These stats helped me understand it better.
[EDIT: I made a mistake in my Currier A/B separation for these plots. Corrected plots in my reply on page 2.]
1. The word length plots for Currier A and B with the distributions for 4 reference languages. (I just chose 4 languages that were easily accessible to me.)
![[Image: word_length_stats.png?raw=true]](https://github.com/Marvel4U/Voynich_semantic_exploration/blob/master/plots/word_length_stats.png?raw=true)
2. The known Zipf distribution of word frequencies with reference languages
![[Image: Zipf_stats.png?raw=true]](https://github.com/Marvel4U/Voynich_semantic_exploration/blob/master/plots/Zipf_stats.png?raw=true)
3. Bigram heatmap.
This one was interesting to me because it shows a very close correspondence between Currier A and B. I expected a much bigger variation.
![[Image: bigram_heatmap_a_b.png]](https://github.com/Marvel4U/Voynich_semantic_exploration/raw/master/plots/bigram_heatmap_a_b.png)
As a reference looked at the bigram statistics of the reference languages and one can see that they vary much much more from each other, compared to the currier A and B.
![[Image: bigram_heatmap_ref_shared.png]](https://github.com/Marvel4U/Voynich_semantic_exploration/raw/master/plots/bigram_heatmap_ref_shared.png)
4. Word start and end bigrams/trigrams
The bigrams and word-initial trigrams did not show that much irregularity but the word-end trigrams clearly shows the famous -edy ending for Currier B
![[Image: word_end_trigrams_a_b.png]](https://github.com/Marvel4U/Voynich_semantic_exploration/raw/master/plots/word_end_trigrams_a_b.png)
What did surprise me is that the -edy ending is also among the most common endings in the Currier A script. From what I read and saw, I assumed that it is almost exclusive to the Currier B. Does that mean that I (a) simply misunderstood or (b) chose the wrong page split between currier A and B?
My current split is defined by the code below. Input is very welcome.
I'll leave it at that for now. I'm curious how the interaction in the community here works and if I'll hear from anybody. I'm still preparing the plots for the TF-IDF analysis, as I said, I actually think they are quite interesting. I will add them as reply when I'm done.
Until then, cheerio,
Marvin.
I've been binging Koen's videos on Youtube over the last week, great stuff!
Obviously that means I'm a Voynich novice, but I did use computational linguistics during my PhD at the MPI in Nijmegen, Netherlands, so I couldn't help but dig my fingers into the data :)
I'm not claiming any novelty but I haven't seen the different analysis steps put together in one place so I figured I might as well publish it here. (However, I do think in the end, I have some interesting results that I didn't see anywhere else... but more about this below and in the next post.)
I put the data, scripts and a small analysis report on a dedicated Github repository, if somebody wants to have a deeper look: You are not allowed to view links. Register or Login to view.
My main idea for this round is to perform an TF-IDF analysis. This is a statistical tool where you count how often words occur over the whole text, vs the individual text segments (pages). With this method, one can (approximately) distinguish "content words" from "function words". Content words are those that are specific to a particular topic, like "Voynich" or "Quantum", while function words are words that show up everywhere, like "the", "of", "and", etc. I think it would be tantalizing to produce a list of Voynich words, where we can guess, from section and illustration cues, what they might mean, given where they show up. (Although I don't think it would bring us closer to deciphering the text, it would be fun.)
Down the line, maybe I have time to produce a visual tool where people can explore how words cluster in certain portions of the text. Not quite as fancy as the amazing tool on voynichese.com but in the same spirit.
I'm currently getting into working with LLMs (building them, not talking to Chat GPT) and I am very curious if one can use these tools to identify semantic clusters of Voynich words. Tbd.
I obviously haven't read everything there is on Voynich, but I did my best to go through voynich.nu and Bowern and Lindemann (2020) as well as the latest and pinned posts on this forum, to get a base understanding what's commonly known and what's currently under discussion. I'm looking forward to learn more from you.
I want to start by stating my base preconceptions/assumptions when I went into the analysis, as well as some questions that maybe you can help me with.
Assumptions:
1. The text is real in the sense that somebody in the 15th century wrote something down to communicate information to somebody else.
2. The transcription is reasonably good and conveys the textual content of the VMS to an overwhelming degree, so we can base an analysis on it.
3. The words are words in the sense that they can, through translation, combination, compression, augmentation by auxiliary information, or some other process, be rendered into a language that someone at some time spoke. If there is a cipher, it did not jumble words by moving word boundaries or similar shenanigans.
4. Letters are only meaningful with regards to the manuscript itself. They cannot be identified in a one-to-one manner with any language.
5. The manuscript was written by several scribes/authors, possibly at different times, possibly without knowing each other. The known separations are Currier A and B as well as the 5 Hands (Davis, Lisa Fagin. 2020).
6. There is no hope of me ever decrypting the text by myself since I have none of the necessary skills to actually understand any language that the authors spoke, even less the manuscript itself.
Questions:
1. The whole analysis is based on IT2a-n.txt from You are not allowed to view links. Register or Login to view. Is this the correct choice? As far as I understand, it's a version of the TT transcription, but I don't know what the state of the art is. I noticed that the transcription used on voynichese.com is different and in some places more complete, but I don't know.
2. How is the interaction between "fan groups" like this one and the academic community? From what I saw in the videos, there is a fairly good collaboration, but I still wonder. I know that for some topics that garner so much public interest, there can be a lot of tension.
3. I had trouble finding a "definitive" distinction of pages into Currier A and B. I don't know if that's because it is not fully defined, if there is disagreement or if I just didn't look at the right places.
Base results:
Before we get to the good stuff, I want to post the base analysis as a sanity check but also so they are all in one spot. As I said, these are all well known but it's good for me to see the data myself, so maybe also for others.
I split up all the analysis steps by Currier A and B. Going in, I did not have any idea how close both languages are. My initial assumption was actually that they are as different as German and Latin. These stats helped me understand it better.
[EDIT: I made a mistake in my Currier A/B separation for these plots. Corrected plots in my reply on page 2.]
1. The word length plots for Currier A and B with the distributions for 4 reference languages. (I just chose 4 languages that were easily accessible to me.)
2. The known Zipf distribution of word frequencies with reference languages
3. Bigram heatmap.
This one was interesting to me because it shows a very close correspondence between Currier A and B. I expected a much bigger variation.
As a reference looked at the bigram statistics of the reference languages and one can see that they vary much much more from each other, compared to the currier A and B.
4. Word start and end bigrams/trigrams
The bigrams and word-initial trigrams did not show that much irregularity but the word-end trigrams clearly shows the famous -edy ending for Currier B
What did surprise me is that the -edy ending is also among the most common endings in the Currier A script. From what I read and saw, I assumed that it is almost exclusive to the Currier B. Does that mean that I (a) simply misunderstood or (b) chose the wrong page split between currier A and B?
My current split is defined by the code below. Input is very welcome.
Code:
CURRIER_A_RANGES = [("f1r", "f24v"), ("f31r", "f31v"), ("f88r", "f90v1"), ("f100r1", "f116r"), ]
CURRIER_A_SINGLES = ["f25r", "f25v", "f32r", "f32v", "f33r", "f34r", "f34v", "f67r2", "f67v1", "f67v2", "f91v"]
CURRIER_B_RANGES = [ ("f26r", "f30v"), ("f35r", "f39v"), ("f75r", "f84v"), ("f93r", "f96v"), ]
CURRIER_B_SINGLES = ["f68r1", "f68r2", "f68v1", "f68v2"]I'll leave it at that for now. I'm curious how the interaction in the community here works and if I'll hear from anybody. I'm still preparing the plots for the TF-IDF analysis, as I said, I actually think they are quite interesting. I will add them as reply when I'm done.
Until then, cheerio,
Marvin.
It looks very interesting.