Independent Analysis Identifying Languages Beyond Currier A/ B, compared to René Zand

[npcompl33t]

It doesn't read too AI to me, which is why I wondered if anyone is able to comment on the research.

I would like that too. I have very broadly understood the method, but most of the details are too much for me and I cannot say anything about the validity of the results. I have to say, they are not easy to understand (maybe a picture would be worth a thousand words...) and to put in context, however they look interesting and I'd like them to be properly assessed.

One random question for @npcompl33t

6. f58
  RZ designation: A
  Divergence from early Herbal A: 100%; from Herbal B: 75%

By f58 do you mean You are not allowed to view links. Register or Login to view. + f58v? And, what's its divergence from the other partitions you identified (Late herbal A + Pharma, Herbal B1 etc.)?

Yes - f58 = You are not allowed to view links. Register or Login to view. + f58v. 

Below is the chart that shows the Jensen-Shannon divergence. Since it may be hard to contextualize these values, i've expressed them as a ratio of the difference of the Early Herbal A section against Herbal B, since that distinction represents the classically recognized Currier A / Currier B split. A 1 means the divergence is equal to the difference between Early Herbal A and Herbal B. 1.5 would mean the divergence is 1.5x greater than the Herbal A/ B difference, 0.5 means it is half as much. 

   

As far as the method, i was trying to keep the post from being bogged down by a lengthy technical discussion, and I've noticed that complex or detailed methodologies will often get flagged as AI. THIS IS NOT AI GENERATED, I WROTE ALL OF THIS.

Most attempts I've seen for analyzing and classifying potentially different 'languages' utilize Principle Component Analysis (PCA) run on bigram counts of pages. This is a perfectly valid technique, but as I mentioned there are a few reasons I wanted to try something different.

Basically I first started by doing analysis to see if I could represent the pages in a way that made the differences more apparent. Rather than comparing the overall frequency of each bigram, I compared the distribution of possible continuations after each preceding unit. This distinguishes how often an “e” occurs from what tends to follow it.  I also used the transcription alphabet proposed by Zattera 2022.

This is fundamentally similar to a bigram, since each row and column together are a bigram, however, it changes how the information is normalized and weighted.

Specifically, it allows us to distinguish how frequent a transition is in contexts where it has the opportunity to appear. For example, a low frequency of ed can be a result of there simply not being any 'e' units, OR it could be because D rarely follows E. Those are two completely different construction patterns that a bigram-only view could obscure.

Mathematically, this is P( d | e) = Number of Transitions from E to D / Total number of transitions out of E

This also includes word start and end as transitions, so initial and end glyphs are considered. This isn't perfect, as it can miss features that are longer than 2 characters, which the bigram approach also misses. I did also analyze 2nd order transitions -- which would capture these, and I did identify one, ^yk (start-y-k) and yka is a marker that splits Stars and Bio. However, it didn't change the picture much.

For each preceding unit, I then compared the distribution of all following possible units using Jensen-Shannon divergence, which is basically a way of measuring how different two distributions are, that is commonly used in natural language processing and information theory.

The main clustering method was Ward hierarchical clustering, using the square root of JS divergence as the distance between texts. I ran this at both page and physical-sheet level. To check reproducibility, I constructed profiles from alternating lines, clustered the two halves separately, and measured agreement between their group assignments using the adjusted Rand index, which is a statistical measure for evaluating the similarity between two clusters, accounting for random chance.

To benchmark how much divergence is normal, i created a metric that estimates how much divergence exists within a single page. I did this by splitting alternating lines on each page into 2 groups, then measuring the JS divergence between those groups.

I then tested various possible subdivisions of these larger groups.

After that, I tested various permutations moving entire physical sheets between groups.

As a cross check, i did the same clustering using Hellinger vectors. Without getting to deep into the math -- this is a different way of measuring how much two populations overlap from Jenson-Shannon divergence. It changes the weighting as well as the distance calculation, so agreement between the two representations provides a useful robustness check.

I also wanted to test for 'outlier' pages that might not be captured by clustering methods that may preferentially create large clusters. To do this, I created the outlier statistic I mentioned in the opening post. Basically, it compares the amount of divergence within a single page, against its closest neighbor. The reverse side of the same physical sheet was excluded from this, You are not allowed to view links. Register or Login to view. was excluded from f58v's comparison. Many pages, when divided in half, have larger divergence than the nearest page. This method identifies single pages that are internally consistent but different from any other page -- both pages of f58 scored extremely high on this metric, with You are not allowed to view links. Register or Login to view. scoring above f58v. 

The final list therefore combines all of the above clustering results, reproducibility checks, comparisons between specified populations, and the separate outlier analysis. It is not simply the output of one clustering run at a chosen number of clusters.