• Options
• Login
• Desktop Style

"The Currier languages revisited" revisited

• Index

  • The Voynich Ninja
  • Voynich Research
  • Analysis of the text
  • "The Currier languages revisited" revisited
• "The Currier languages revisited" revisited

• "The Currier languages revisited" revisited

  kckluge > Yesterday, 04:05 AM

  One page on Rene's site (You are not allowed to view links. Register or Login to view.) does a bigram frequency level analysis of the pages in the manuscript. This post is specifically addressing the section starting with "Language characteristics". The analysis on that part of the page:
  
  * uses Rene's CUVA alphabet to deal with EVA's oversegementation of the glyphs (You are not allowed to view links. Register or Login to view.)
  
  * removes uncertain spaces from the transcription, but leaves other spaces
  
  * only looks at bigrams within words, not bigrams straddling spaces
  
  * starts with a feature space corresponding to the relative frequencies of all 355 CUVA bigrams that occur, then does a dimensionality reduction similar (but not identical) to Principle Components Analysis (PCA -- You are not allowed to view links. Register or Login to view. describes PCA)
  
  Plots are shown for the dominant vector vs. the 2nd through 4th vectors found by his method. On the basis of those plots he concludes, "When Currier identified his languages A and B, he did this on the basis of the different statistics of the initial herbal pages in the MS, which are identified by the red ('A') and dark blue ('B') crosses. It is clear that these have distinct properties - the clouds do not overlap. He also checked the other pages, and noted more variations, but his criteria for distinguishing the languages did not allow him to see that the overall statistics demonstrate that there is a continuum, and the other (not herbal) pages actually 'bridge the gap'."
  
  It is important to be careful about drawing conclusions from linear projections of higher dimensional data onto lower dimensional spaces. If two clumps of points are separable in the lower dimensional projection then they are also separable in the full dimensional space, but the inverse is not true -- two clumps of points that overlap in some projection do not necessarily overlap in the full space. 
  
  To examine Rene's conclusion I performed a variation of the analysis described above:
  
  * the Currier alphabet is used rather than CUVA, translated from the ZL_ivtff_1b.txt EVA transcription (when multiple proposed reading are given for a glyph the first option is used)
  
  * uncertain spaces are removed as per the original experiment
  
  * only lines corresponding to running paragraph and "circular" text -- no radial text from diagrams or labels
  
  * only the 40 most common bigrams are used -- in Currier these are:
  
  89 OF OE 4O CC C8 SC 8A C9 AM FC OP CO AR FA AE OR ZC SO O8 PC AN PA EF FS ZO PS S9 ES RA S8 9F AJ BS F9 FO PO 2A 9P EO
  
  which correspond to EVA
  
  dy ok ol qo ee ed che da ey aiin ke ot eo ar ka al or she cho od te ain ta lk kch sho tch chy lch ra chd yk am pch ky ko to sa yt lo
  
  * bigrams including spaces (with end-of-line, end-of-paragraph, and plant drawing gaps counted as spaces) are included in the total bigram count for a page when computing relative bigram frequencies for the page
  
  The 40 Currier bigrams listed above cover 83% of the bigrams that don't include a space or untranslatable/transcribed non-Currier "wierdo". Applying PCA, the first two dimensions found capture 48% of the covariance in the 40-D data. The resulting plot is:
  
     
  
  With the exception of three pages, the Herbal B, Bio, Starred paragraph, and Rose foldout pages fall together in one cluster and the Herbal A, Astro, Zodiac, and Pharma pages fall together in another cluster, separated by a clear diagonal gap. 
  
  One exception is Zodiac page f73v; the other two exceptions are You are not allowed to view links. Register or Login to view. & f65v. f58 & f65 are the halves of a biofolio that Lisa Fagin-Davis identifies as by Scribe 3; You are not allowed to view links. Register or Login to view. has a plant drawing with no text other than a 2-3 word label. Traditionally those bifolio pages have been labelled as A Language, which would make this the only known non-Scribe 1 Herbal A biofolio. It is plausible that the f58 & f65 bifolio pages are B language pages with atypical relative frequencies of the small number of key bigrams used to make the initial A/B classification by Currier (in which case You are not allowed to view links. Register or Login to view. becomes the outlier grouped with the Herbal A pages).
  
  The differences between the analyses are:
  
  * use of CUVA vs Currier
  
  * inclusion of radial and label text elements vs only running paragraph and "circular" diagram text
  
  * starts with a 355-D space (all bigram frequencies) vs a 40-D space (only most common, corresponding to 83% of the glyph bigram pairs in the text)
  
  * dimensionality reduction using a heuristic PCA-like method rather than PCA
  
  The lack of clear separation between the A and B languages in Rene's plots is most likely due to a combination of very low frequency bigrams adding noise into the data with suboptimal choice of basis vectors by his dimensionality reduction method.

• RE: "The Currier languages revisited" revisited

  Koen G > Yesterday, 10:55 AM

  Very interesting and comprehensible analysis, thank you. Quire 8 pages misbehaving comes as no surprise.
  
  I wonder if the cloud of Zodiac pages can be seen as somewhat bridging the gap? Maybe it behaves differently because it's purely circular text?
  Likely, which pages of Pharma and Stars are part of the Herbal A cloud? Might this also have something to do with the ratio of paragraph text vs. labels/circular text?

• RE: "The Currier languages revisited" revisited

  MarcoP > Yesterday, 01:19 PM

  Quote:dimensionality reduction using a heuristic PCA-like method rather than PCA

  
  Why not using PCA? That would make for a more meaningful comparison with Rene's results

• RE: "The Currier languages revisited" revisited

  kckluge > 6 hours ago

  (Yesterday, 01:19 PM)MarcoP Wrote: You are not allowed to view links. Register or Login to view.

  Quote:dimensionality reduction using a heuristic PCA-like method rather than PCA

  
  Why not using PCA? That would make for a more meaningful comparison with Rene's results

  
  I was ("Applying PCA, the first two dimensions found capture 48% of the covariance in the 40-D data."). I should probably have done that as a table to avoid ambiguity. What Rene did is described on his page as follows:
  
  

  Quote:The following procedure will not necessarily find this maximum, but it will find something near to the maximum.
  

  1. Locate the bigram which has the most varying distribution over all pages. This means: locate the component of all vectors which has the largest rms about mean.
     
  2. For all bigrams, find out their covariance with this bigram. This means, estimate a linear relationship (without offset) which best fits the value of each component vs. the value of the component found above. Save the slope of this linear relationship.
     
  3. The vector containing all these slopes has the value 1 at the component found above, and values between -1 and 1 at all other components. Normalise this vector and the base vector for the prime direction has been found.
     

  Now, the coefficient along this base vector may be computed for each page vector. The contribution of this base vector may then be subtracted from each of the page vector, meaning that the hypercloud collapses to a space with dimension one less than before.
  After this last step, the procedure may be repeated, and it will automatically find the next most important base vector, which will be perpendicular to the first. This whole procedure may be repeated several times.

• RE: "The Currier languages revisited" revisited

  kckluge > 4 hours ago

  (Yesterday, 10:55 AM)Koen G Wrote: You are not allowed to view links. Register or Login to view.Very interesting and comprehensible analysis, thank you. Quire 8 pages misbehaving comes as no surprise.
  
  I wonder if the cloud of Zodiac pages can be seen as somewhat bridging the gap? Maybe it behaves differently because it's purely circular text?
  Likely, which pages of Pharma and Stars are part of the Herbal A cloud? Might this also have something to do with the ratio of paragraph text vs. labels/circular text?

  
  If only running paragraph text is used with the same 40 bigrams, the results are similar with a couple caveats: f70r2 is the only Zodiac page in the sample; several of the Astro/Cosmicological pages drop out due to lack of running paragraph text; and the text sample sizes on the diagram pages get smaller (and therefore noisier from a variance perspective). Here these two axes capture a hair over 50% of the overall covariance:
  
     
  
  The f58 & f65 pages still fall in the B cluster (unsurprisingly). The lone Zodiac page is on the border between the A & B clusters, and one of the Rose foldout pages drifts towards the big Pharma group, but that's as likely to be due to the smaller text sample on those pages after circular text is excluded as anything else.
  
  With respect to the relationship between the Herbal A, Pharma, Zodiac, and Astro (circular diagram) pages, using the same 40 bigrams on running paragraph and circular text but doing PCA on just those data points yields the following plot (first two dimensions capture ~43.5% of the total covariance of that set of points):
  
     
  
  While there is some overlap, it looks like the zodiac & circular diagram pages are in a different "dialect" than the pharma & pharma-adjacent herbal pages. That's why it's so important to be careful about interpreting lower dimensional projections -- they look like they fall on top of each other when you use the first two axes returned by PCA over all the pages, but that's an artifact of PCA being forced to find directions that capture the main A/B split.
  
  Doing the same thing for the B language pages (the first two axes here capture ~48.3% of the B page covariance), the Bio pages fall in a group towards the upper right; the Herbal B pages (albeit with a few stragglers), the Rose foldout panels, and a chunk of the Starred paragraph pages fall in a group towards center left, and a chunk of the Starred paragraph pages fall in between the two (presumably reflecting the split between Starred paragraph pages Rene found on the basis of differences in most common word vocabulary -- You are not allowed to view links. Register or Login to view.):
  
     
  
  The same caveat about caution interpreting overlaps in a lower dimensional projection apply -- pairwise comparisons of Herbal B, Bio, Rose foldout, and Starred paragraph sets of pages would probably give a more reliable sense of the substructure in the B language "dialects".
  
  With respect to "if the cloud of Zodiac pages can be seen as somewhat bridging the gap", the point of the analysis was to test (and push back on in a data driven way) the whole notion that there is some smooth gradual transition between the A language pages and the B language pages. That's just not what the relative bigram frequency data appears to be saying. While they are probably (almost certainly) instances of some common underlying scheme, they appear to reflect separate discrete versions of that scheme.
  
  An interesting question is what the PCA plots would look like if tested on the 40 most common bigrams in comparably sized "pages" of mixed (say) English, French, and Spanish. I suspect the scatter within a given language would be lower and rather more Gaussian looking than the results for the Voynich pages.

• RE: "The Currier languages revisited" revisited

  ReneZ > 3 hours ago

  Thanks Karl, for this additional analysis. I find these types of cross-verification very useful, and one of the main purposes of having standardised formats is precisely to allow this. This was a major component of some of the work in my professional life, and the parallel processing of the same or similar data by several independent groups was one of the main reasons for the spectacular improvement in the results that could be achieved over a few decades.
  
  [ Unfortunately, that is entirely off-topic here :-) ]
   
  Anyway, there is quite a lot that can be said about the commonalities and differences, and for much of it, additional experiments would have to be run. That is something I am not set up to do on short notice, and will have to stay a bit further down on a list.....
  
  So, the following are some educated guesses.
  
  1. Currier vs. Cuva: my guess is that this is not likely to cause any significant difference to the result, perhaps not even visible. This could be tried out exactly.
  
  For this, note that the results on the linked page were done in 1999 or so, but I repeated them before the 2022 conference, using somewhat different input data, and a slightly different definition of Cuva. I also tried 'real' PCA vs. my own invented logic. All this made hardly any difference to the result.
  
  Unfortunately, and as chance will have it, I recently discontinued the page that had these results (You are not allowed to view links. Register or Login to view.).
  I can temporarily bring it back if there is an interest - it may have formatting issues due to changes I made to the overall site formatting.
  
  2. Using only the most common 40 bigrams vs all: here I also believe that this is not likely to cause any significant or even visible difference to the result. To test this would be slightly more complicated.
  
  3. Not using the text on circles and labels: I strongly suspect that this is the main reason for the visible differences. I especially suspect the circular texts. It may be worth to try this out.
  
  I showed on the original page that leaving out the foldout pages creates a major gap in the whole cluster. Now of course this is not the same thing, but still almost all circular text is on foldout pages, so there is some hint for a relationship.
  
  

  (Yesterday, 04:05 AM)kckluge Wrote: You are not allowed to view links. Register or Login to view.It is important to be careful about drawing conclusions from linear projections of higher dimensional data onto lower dimensional spaces. If two clumps of points are separable in the lower dimensional projection then they are also separable in the full dimensional space, but the inverse is not true -- two clumps of points that overlap in some projection do not necessarily overlap in the full space. 

  
  I fully agree with this, but I would argue as follows:
  The projections I used to map the higher-dimensional space onto individual planes were chosen such that the extent (maximum minus minimum) of each dimension is sorted from high to low. The size of the cloud along each later dimension becomes uniformly smaller. Chances that there is a clean separation in a later dimension become smaller and smaller.
  
  Of course, reality is more complicated. A separation does not have to be along a plane, and one should really be able to rotate the cloud freely to look at it from more angles to check this. However, I see no indication in the plots I have made, that there is a separation. 
  
  Finally, there is also this point: while Currier posited two different languages, intermediate forms (which he had not seen or at least not mentioned) exist. However, the amount of text of these intermediate forms is clearly less than both the 'clearly' A and 'clearly' B material. This also affects the appearance of the point clouds.
  
  It is interesting to see that you seem to have three clouds, i.e. not just A and B.
  
  The overall 'boomerang' shape is also there. I think that this should be attributed to the fact that all coordinates are values between 0 and 1, where most of them are close to 0 , and some stretch out along one of the axes.
  If the point cloud were not projected along Eigenvectors, but along the base vectors, they are all dominated by points along the two axes.
• Next Oldest Next Newest