+- The Voynich Ninja (https://www.voynich.ninja)
+-- Forum: Voynich Research (https://www.voynich.ninja/forum-27.html)
+--- Forum: Analysis of the text (https://www.voynich.ninja/forum-41.html)
+--- Thread: Independent Analysis Identifying Languages Beyond Currier A/ B, compared to René Zand (/thread-6084.html)
Pages:
1
2
RE: Independent Analysis Identifying Languages Beyond Currier A/ B, compared to René Zand - npcompl33t - 18-09-2026
(18-09-2026, 10:20 PM)nablator wrote: You are not allowed to view links. Register or Login to view.(17-09-2026, 01:01 AM)npcompl33t wrote: You are not allowed to view links. Register or Login to view.The second is that the selected bigrams could completely miss large outliers, for example, as is the case with f58.
The frequency of "al" alone should set f58 apart: You are not allowed to view links. Register or Login to view.
Nick Pelling wrote about the peculiarities of f58: You are not allowed to view links. Register or Login to view.
Yes, I 100% agree, which is why I was surprised that Rene didn't identify it despite using -al as a part of his classification scheme for his 'C' grouping.
I have read that thread before and in fact, it is one of the reasons I decided to make this post at all. The thread's premise is that f58 is an 'intermediate' language, and marco P shows the PCA bigram chart that seems to show f58 in the middle of A and B sections.
However, that chart is showing only the 1st and 2nd dimensions, which together only capture something like 20% of the total variance. The issue is that the PCA is attempting to explain variance across the entire manuscript, and since f58 is only 2 pages, the features that distinguish it don't rank as high. Checking the 3rd dimension alone shows that it is not really 'in between' A and B, but rather off on its own.
RE: Independent Analysis Identifying Languages Beyond Currier A/ B, compared to René Zand - Mauro - 19-09-2026
(18-09-2026, 10:11 PM)Koen G wrote: You are not allowed to view links. Register or Login to view.It doesn't read too AI to me, which is why I wondered if anyone is able to comment on the research.
I would like that too. I have very broadly understood the method, but most of the details are too much for me and I cannot say anything about the validity of the results. I have to say, they are not easy to understand (maybe a picture would be worth a thousand words...) and to put in context, however they look interesting and I'd like them to be properly assessed.
One random question for @npcompl33t
Quote:6. f58
RZ designation: A
Divergence from early Herbal A: 100%; from Herbal B: 75%
By f58 do you mean You are not allowed to view links. Register or Login to view. + f58v? And, what's its divergence from the other partitions you identified (Late herbal A + Pharma, Herbal B1 etc.)?
RE: Independent Analysis Identifying Languages Beyond Currier A/ B, compared to René Zand - npcompl33t - 21-09-2026
(19-09-2026, 04:58 PM)Mauro wrote: You are not allowed to view links. Register or Login to view.(18-09-2026, 10:11 PM)Koen G wrote: You are not allowed to view links. Register or Login to view.It doesn't read too AI to me, which is why I wondered if anyone is able to comment on the research.
I would like that too. I have very broadly understood the method, but most of the details are too much for me and I cannot say anything about the validity of the results. I have to say, they are not easy to understand (maybe a picture would be worth a thousand words...) and to put in context, however they look interesting and I'd like them to be properly assessed.
One random question for @npcompl33t
Quote:6. f58
RZ designation: A
Divergence from early Herbal A: 100%; from Herbal B: 75%
By f58 do you mean You are not allowed to view links. Register or Login to view. + f58v? And, what's its divergence from the other partitions you identified (Late herbal A + Pharma, Herbal B1 etc.)?
Yes - f58 = You are not allowed to view links. Register or Login to view. + f58v.
Below is the chart that shows the Jensen-Shannon divergence. Since it may be hard to contextualize these values, i've expressed them as a ratio of the difference of the Early Herbal A section against Herbal B, since that distinction represents the classically recognized Currier A / Currier B split. A 1 means the divergence is equal to the difference between Early Herbal A and Herbal B. 1.5 would mean the divergence is 1.5x greater than the Herbal A/ B difference, 0.5 means it is half as much.
As far as the method, i was trying to keep the post from being bogged down by a lengthy technical discussion, and I've noticed that complex or detailed methodologies will often get flagged as AI. THIS IS NOT AI GENERATED, I WROTE ALL OF THIS.
Most attempts I've seen for analyzing and classifying potentially different 'languages' utilize Principle Component Analysis (PCA) run on bigram counts of pages. This is a perfectly valid technique, but as I mentioned there are a few reasons I wanted to try something different.
Basically I first started by doing analysis to see if I could represent the pages in a way that made the differences more apparent. Rather than comparing the overall frequency of each bigram, I compared the distribution of possible continuations after each preceding unit. This distinguishes how often an “e” occurs from what tends to follow it. I also used the transcription alphabet proposed by Zattera 2022.
This is fundamentally similar to a bigram, since each row and column together are a bigram, however, it changes how the information is normalized and weighted.
Specifically, it allows us to distinguish how frequent a transition is in contexts where it has the opportunity to appear. For example, a low frequency of ed can be a result of there simply not being any 'e' units, OR it could be because D rarely follows E. Those are two completely different construction patterns that a bigram-only view could obscure.
Mathematically, this is P( d | e) = Number of Transitions from E to D / Total number of transitions out of E
This also includes word start and end as transitions, so initial and end glyphs are considered. This isn't perfect, as it can miss features that are longer than 2 characters, which the bigram approach also misses. I did also analyze 2nd order transitions -- which would capture these, and I did identify one, ^yk (start-y-k) and yka is a marker that splits Stars and Bio. However, it didn't change the picture much.
For each preceding unit, I then compared the distribution of all following possible units using Jensen-Shannon divergence, which is basically a way of measuring how different two distributions are, that is commonly used in natural language processing and information theory.
The main clustering method was Ward hierarchical clustering, using the square root of JS divergence as the distance between texts. I ran this at both page and physical-sheet level. To check reproducibility, I constructed profiles from alternating lines, clustered the two halves separately, and measured agreement between their group assignments using the adjusted Rand index, which is a statistical measure for evaluating the similarity between two clusters, accounting for random chance.
To benchmark how much divergence is normal, i created a metric that estimates how much divergence exists within a single page. I did this by splitting alternating lines on each page into 2 groups, then measuring the JS divergence between those groups.
I then tested various possible subdivisions of these larger groups.
After that, I tested various permutations moving entire physical sheets between groups.
As a cross check, i did the same clustering using Hellinger vectors. Without getting to deep into the math -- this is a different way of measuring how much two populations overlap from Jenson-Shannon divergence. It changes the weighting as well as the distance calculation, so agreement between the two representations provides a useful robustness check.
I also wanted to test for 'outlier' pages that might not be captured by clustering methods that may preferentially create large clusters. To do this, I created the outlier statistic I mentioned in the opening post. Basically, it compares the amount of divergence within a single page, against its closest neighbor. The reverse side of the same physical sheet was excluded from this, You are not allowed to view links. Register or Login to view. was excluded from f58v's comparison. Many pages, when divided in half, have larger divergence than the nearest page. This method identifies single pages that are internally consistent but different from any other page -- both pages of f58 scored extremely high on this metric, with You are not allowed to view links. Register or Login to view. scoring above f58v.
The final list therefore combines all of the above clustering results, reproducibility checks, comparisons between specified populations, and the separate outlier analysis. It is not simply the output of one clustering run at a chosen number of clusters.
RE: Independent Analysis Identifying Languages Beyond Currier A/ B, compared to René Zand - npcompl33t - 07-10-2026
I have been pushing this project ahead alongside some of my other work, and have some updates to share.
Web App / Git Repo
First, i have created You are not allowed to view links. Register or Login to view. (You are not allowed to view links. Register or Login to view.)that allows you to fully examine all page classifications against RZ's, Curriers, and LFD hands. You can also view the entire page level JS divergence scores, and outlier scores. Note, this classification scheme is referred to as 'Golf Regimes' in the app, this is my internal name, resulting from my habit of naming workstreams Alpha, Bravo, Charlie, etc.
Page-level RZ Comparison
RZ's classifications contain quite a bit more 'noise' than I thought from reading his write-up alone. The You are not allowed to view links. Register or Login to view. between our 2 clusters is 0.609, quite a bit lower than I had originally assumed. To be fair, his write-up says this is preliminary and he knows of several improvements he wants to make to his system, so the comparison below is for people to understand the current differences only.
I dug into some of the more obvious discrepancies, like the scattered Herbal A pages he groups into 'C', which his write-up labels 'Cosmological' but appears to mostly be a residual category with minimal cosmological content.
About half of these appear to be cases that are right on the edge of his classification system. For example,
- he places You are not allowed to view links. Register or Login to view. into C because 2/56 of its word contain ed. If it lost one of those, it would fall into the A category. f27 is an odd page i will discuss more below.
- f113r he places in Ce, at 12%, it just misses the B cutoff, and eo at 16.8% is just above the e-dialect cutoff. One additional ed word would push this into Bs
The other half appear to result from his hierarchical clustering system operating on a few rules:
- f80r, his single Cb classification, is because its ed is 10.8%, but it satisfies the Bio markers, creating its own category
- f116r satisfies markers for two of his classifications, but bio takes priority, so it gets put there instead of stars
- f107r -> C its ed share is 9%, even though it has strong stars markers, it cannot be placed there because its 'ed' percentage makes its main letter C
I'm not sure how these could be fixed, with just a few rules driving a rigid hierarchical clustering, slightly changing any of these would likely create other exceptions. It would probably require introduction of more features into the classification scheme.
JS Divergence Cluster Map
The website i shared above allows you to view the JS divergence between each pages word level transition matrix. Again, my method creates a glyph transition matrix, like start->ch -> o -> l -> end, and then calculates the JS divergence between that page and other pages. The central line represents the pages internal JS divergence, that is, if you split the page in half, and measure the JS divergence between both halves.
You can fully zoom and explore this chart in the app, below is a screenshot. In addition to viewing this for my clusters, you can view normal manuscript order, grouped by bifolium, and RZ's clusters.
New Possible Subdivisions
Close analysis of this matrix can reveal some interesting possible smaller subdivisions. For example, There is a smaller but noticeable difference between pharma and late Herbal A.
Early Herbal A also appears to have a noticeable split. I identified this previously, but it is not clear if it represents a continuum or two distinct clusters. These can be most easily separated by comparing the frequency of words beginning with cho* and words beginning with y*. The colors in the chart below are the categories assigned by ward clustering on the page level JS divergence.
A similar subdivision exists within stars. The splits are roughly equal in magnitude (i.e. total divergence between the groups), but the stars group separates more cleanly.
In both cases, these subdivisions are not as significant as the top level 'regimes'. I'm considering extending my classifications into more subdivisions to capture these differences.
You may notice other smaller groupings in the app's cluster-map, if you see something and are curious, just ask and i can look into it.
Improved Outlier Detection
I made two updates to my 'outlier' detection. Basically my clustering method attempts to create large groups, so it has difficulty with single page isolate 'outliers'. As a result, i created the 'outlier' detection mentioned in my OP. It involves measuring the JS divergence between two halves on one page, and then comparing that to its nearest neighbor, excluding other pages in the same bifolium.
This would manifest on the cluster map as mostly isolated darker lines.
This had a few possible shortcomings:
- first, i only ran my initial clustering algorithm on prose text, so some of the pages that are mostly labels or other loci were excluded. Expanding to include all loci, for instance, causes fRos (rosettes) to be the 3rd largest outlier, although still significantly below f58r/v. The app supports both versions, you can toggle between them with a selector.
- second was, this method can miss outliers that consist of two pages in different bifolium. They would each map to each other in the nearest neighbor search, and produce a low divergence score, even though the 2nd closest neighbor is actually far away, and the pair is isolated.
- Finally, probably the most significant, is how the outlier score is created. The outlier score is essentially the ratio of (how different is the page from nearest non-bifolium neighbor) / (how much difference exists within the page). f58r, at 1.7x, means the JS divergence to its nearest neighbor is almost double its internal variation. However, this method can have trouble with pages with low amounts of text, since the amount of text is extremely inversely correlated to internal divergence, -.8. This is somewhat expected, with less text, it is simply harder to know for sure if a page is an outlier or it is just a sampling artifact.
The chart below shows internal JS divergence (how much variance exists within the page) on the Y axis, and Nearest 2nd neighbor JS on the right axis, using all types of text loci (labels, spherical, etc). The full interactable chart is at the bottom of the You are not allowed to view links. Register or Login to view..
You can see f58r, f58v, and rosettes, are all cleanly isolated from the main group. These pages all have low internal variance, compared to how different their nearest neighbor is, which is why they have a high outlier score.
Looking to the right side of the graph though, shows a scattering of points that mostly all contain small amounts of text, including our friend You are not allowed to view links. Register or Login to view. I mentioned earlier, along with f57v.
To help to better understand if these divergence scores are just an artifact of a low amount of text, or genuinely seem to be different, i compared them against other similarly small batches of text from the same section. This confirms that many of these indeed are isolates. The full chart is also on the web-app.
- f27v is an odd page, for instance, it is the only page in the entire manuscript to not contain the sequence 'aiin' in any word, and several of its other character statistics differ quite a lot from other Early Herbal A pages. It is in the 99th percentile on this new short-page outlier metric.
- f57v shouldn't be a surprise, even after removing the singletons, the remaining 68 words are still unusual for its length
- f68v2 >99th percentile, but it lacks much comparable celestial material, tough to say
- f65v > 95th percentile, still elevated, but less decisive
- f96v > 99th percentile against other herbals. This is a strong candidate as another singleton outlier
For contrast, You are not allowed to view links. Register or Login to view. lands around the 50th percentile within Early A, and You are not allowed to view links. Register or Login to view. around the 2nd–4th. This method does distinguish ordinary short pages from unusually isolated ones.
RE: Independent Analysis Identifying Languages Beyond Currier A/ B, compared to René Zand - npcompl33t - 10-10-2026
I finished an updated analysis on this, and expanded the classification scheme down another level. At this level, I don' think its fair to refer to these differences as 'languages', 'dialects', etc. But there is obviously still observable glyph level statistics that would be odd in a normal language, and that can confound certain analysis that are focused on glyph-level traits.
Below are these classifications, and you can actually explore the different transitions/bigrams that help distinguish them You are not allowed to view links. Register or Login to view..
A few things to note:
- The traits that the lowest levels represent are mostly gradients, or a continuum, not binary. This clustering separates them, but these are more akin to overlapping distributions, or a single spectrum, than distinct populations.
- The later sections are generally more reliably differentiated more reliably due to having more text. Many of the herbal pages simply don't have enough text, and their internal divergence is high enough, that it is hard to reliably put individual sheets into different groups. Pooling at the bifolio level does show obvious differences, because most pages in a bifolio agree, but individual sheets are harder to say for certain.
- I have tagged which pages are firmly within the distinct cluster, which are tentative, and which have low confidence.
- When splitting stars in half by the method used in my previous post, it turns out each half more closely resembles either Bio or the other Herbal B sections rather than forming a distinct node.
- The most interesting finding, by far, is that across virtually all pages in the manuscript, ke/te and ka/ta are strongly anticorrelated, although the exact level tends to form a gradient. You are not allowed to view links. Register or Login to view.. Bio and Stars as a whole, ka/ta and ke/te are below -.7 correlated, which is so extreme im certain people here have already caught this.
The web app allows you to select any level and view what differentiates the leaves on that branch, along with charts like the following, so see it for a full breakdown. You are not allowed to view links. Register or Login to view., and it contains information about confidence level, as well as notes in cases where judgement calls were made.
You may ask why such categories are helpful, and in some cases, they may not be. However, it is obvious that there are still significant differences captured by this grouping. For example, Early Herbal A is split into 1a and 1b. In 1a, words that start with ch* occur at nearly 2x the rate as 1b. Words like chol and shol drop by about 50% between 1a and 1b. I'm very interested in why chol occurs more often in Herbal A than in Currier B sections, and the fact it occurs 2x as often on some as others is an important clue.
Looking back at RZ's classifications, its clear some of his previous 'outlier' pages are actually picking up on the same features I'm identifying here.
For example, in17v we both detect a Pharma-like departure from Early A. In his You are not allowed to view links. Register or Login to view., he speculated that Herbal B and Stars might split into two groups, as they have here, and his classification features partially recover some of the differences, but sometimes the other rules end up suppressing these signals. For example, the two Herbal B clusters straddle RZ's 12.5% 'ed' C/B cutoff, and his scheme partial recovers that by splitting the group among C/B tags. His scheme almost recovers the Stars split this way as well, as the two groups sit at 15.30%/29.49%, just above the 12.5% cutoff.
RE: Independent Analysis Identifying Languages Beyond Currier A/ B, compared to René Zand - npcompl33t - 11-10-2026
I was actually able to improve this even more, particularly with respect to the Early Herbal A. The primary transitions that were driving these were word start->first glyph. I tried concatenating the entire line together into a single word, then using the entire line transition profile instead of just a word. This appeared to produce different but valid features, so I ended up combining both the line and the word transition matrixes together.
This produces a much cleaner split among Currier A groups. There are clearly 2 distinct populations of Early Herbal A, one 'Transition' Herbal A, and Pharma. This also resolves a few of the other tentative or uncertain points. This produces the following taxonomy.
We can obtain something like the PCA bigram charts. MDS is essentially similar to PCA but for pairwise distance matrixes, which is exactly what the cluster heatmap on the linked site is. It produces the following chart, which has the same rough shape as the PCA bigram chart, but with much better separation of components.
You can actually explore this MDS view fully on the web-app. Clicking on shows what the same MDS dimensions look like on that subset. This helps view the differences between certain areas, performing the MDS on just that section compacts information that is relevant to those sections into the primary 1st and 2nd dimension.
For instance, this shows the MDS split among A pages. You can see exactly what i described: 2 clear poles of Early Herbal A, and pharma form distinct clusters, with a 4th appearing spread out in between pharma and Early Herbal A.
Below is the same MDS chart, showing the progression from the first version to the newest "2.1" version. For 2.0
Links:
You are not allowed to view links. Register or Login to view.
You are not allowed to view links. Register or Login to view.
You are not allowed to view links. Register or Login to view.
RE: Independent Analysis Identifying Languages Beyond Currier A/ B, compared to René Zand - DG97EEB - 11-10-2026
I did something similar with glyphs. I used a SAM model to isolate individual glyphs across the whole Manuscript and created a 1.7mn image dataset. Part of that can be seen here and it shows the same horseshoe pattern .. I have absolutely no idea what do to do with this but if useful, I can share it.. You are not allowed to view links. Register or Login to view. - it's missing bifolium because I did it months ago before Lisa and Colin's paper..
RE: Independent Analysis Identifying Languages Beyond Currier A/ B, compared to René Zand - npcompl33t - 12-10-2026
(11-10-2026, 09:59 AM)DG97EEB wrote: You are not allowed to view links. Register or Login to view.I did something similar with glyphs. I used a SAM model to isolate individual glyphs across the whole Manuscript and created a 1.7mn image dataset. Part of that can be seen here and it shows the same horseshoe pattern .. I have absolutely no idea what do to do with this but if useful, I can share it.. You are not allowed to view links. Register or Login to view. - it's missing bifolium because I did it months ago before Lisa and Colin's paper..
Whoa! You are right, look at that horseshoe! This is incredibly interesting, I would love to dig into this more. Can you share more about how you generated this data and how i might be able to replicate it?
RE: Independent Analysis Identifying Languages Beyond Currier A/ B, compared to René Zand - npcompl33t - 12-10-2026
(11-10-2026, 09:59 AM)DG97EEB wrote: You are not allowed to view links. Register or Login to view.I did something similar with glyphs. I used a SAM model to isolate individual glyphs across the whole Manuscript and created a 1.7mn image dataset. Part of that can be seen here and it shows the same horseshoe pattern .. I have absolutely no idea what do to do with this but if useful, I can share it.. You are not allowed to view links. Register or Login to view. - it's missing bifolium because I did it months ago before Lisa and Colin's paper..
Whoa! You are right, look at that horseshoe! This is incredibly interesting, I would love to dig into this more. Can you share more about how you generated this data and how i might be able to replicate it?