"The Currier languages revisited" revisited

[ReneZ]

Thanks Karl, for this additional analysis. I find these types of cross-verification very useful, and one of the main purposes of having standardised formats is precisely to allow this. This was a major component of some of the work in my professional life, and the parallel processing of the same or similar data by several independent groups was one of the main reasons for the spectacular improvement in the results that could be achieved over a few decades.

[ Unfortunately, that is entirely off-topic here :-) ]
 
Anyway, there is quite a lot that can be said about the commonalities and differences, and for much of it, additional experiments would have to be run. That is something I am not set up to do on short notice, and will have to stay a bit further down on a list.....

So, the following are some educated guesses.

1. Currier vs. Cuva: my guess is that this is not likely to cause any significant difference to the result, perhaps not even visible. This could be tried out exactly.

For this, note that the results on the linked page were done in 1999 or so, but I repeated them before the 2022 conference, using somewhat different input data, and a slightly different definition of Cuva. I also tried 'real' PCA vs. my own invented logic. All this made hardly any difference to the result.

Unfortunately, and as chance will have it, I recently discontinued the page that had these results (You are not allowed to view links. Register or Login to view.).
I can temporarily bring it back if there is an interest - it may have formatting issues due to changes I made to the overall site formatting.

2. Using only the most common 40 bigrams vs all: here I also believe that this is not likely to cause any significant or even visible difference to the result. To test this would be slightly more complicated.

3. Not using the text on circles and labels: I strongly suspect that this is the main reason for the visible differences. I especially suspect the circular texts. It may be worth to try this out.

I showed on the original page that leaving out the foldout pages creates a major gap in the whole cluster. Now of course this is not the same thing, but still almost all circular text is on foldout pages, so there is some hint for a relationship.

It is important to be careful about drawing conclusions from linear projections of higher dimensional data onto lower dimensional spaces. If two clumps of points are separable in the lower dimensional projection then they are also separable in the full dimensional space, but the inverse is not true -- two clumps of points that overlap in some projection do not necessarily overlap in the full space. 

I fully agree with this, but I would argue as follows:
The projections I used to map the higher-dimensional space onto individual planes were chosen such that the extent (maximum minus minimum) of each dimension is sorted from high to low. The size of the cloud along each later dimension becomes uniformly smaller. Chances that there is a clean separation in a later dimension become smaller and smaller.

Of course, reality is more complicated. A separation does not have to be along a plane, and one should really be able to rotate the cloud freely to look at it from more angles to check this. However, I see no indication in the plots I have made, that there is a separation. 

Finally, there is also this point: while Currier posited two different languages, intermediate forms (which he had not seen or at least not mentioned) exist. However, the amount of text of these intermediate forms is clearly less than both the 'clearly' A and 'clearly' B material. This also affects the appearance of the point clouds.

It is interesting to see that you seem to have three clouds, i.e. not just A and B.

The overall 'boomerang' shape is also there. I think that this should be attributed to the fact that all coordinates are values between 0 and 1, where most of them are close to 0 , and some stretch out along one of the axes.
If the point cloud were not projected along Eigenvectors, but along the base vectors, they are all dominated by points along the two axes.