[Article] A new Timm & Schinner publication regarding the Malta conference

[Torsten]

The problem is that when you say, "Let us, for the moment, assume two well-separated domains, A and B." there is an extent to which that is setting up a strawman. It is entirely possible to have multiple populations with overlapping tails in some feature space; the existence of overlapping tails isn't evidence against the existence of multiple underlying distributions.

Obviously we did not quite get your point before (maybe subconsciously caused by the slightly less than polite wording). The absence of a "domain boundary" in our figure 1 was never meant to serve as sufficient condition for the non-existence of two clearly separated text domains, but rather as necessary condition. Our proposed continuous A-B evolution is not based on this result (see our 2020 paper). In hindsight, we should have emphasized this more clearly in the text.

Nevertheless, the methodology behind direct inspection of the cosine distance is not complete garbage (despite its undoubted limitations). In your counter-example with the two bivariate normal distributions you have separated both subpopulations by approximately the maximum possible to ensure that still the effect is no longer visible.

   

Note that a separation by 4σ (as in your example) to 5σ also approximately defines the boundary where the hypothesis of a single population becomes statistically highly insignificant. The VMS A/A, B/B, and A/B distributions show a much bigger overlap (according to your graph), so any separation effects are clearly under the sensitivity threshold of the cosine distance method. Unfortunately, the underlying actual distributions (and their corresponding metric distances) are not trivial to evaluate in this case, and yes, we are aware of the crux of small sample sizes.

Interestingly, this method most likely still produces results beyond random noise when applied to the VMS. In our 2020 paper we proposed a sorting order of the VMS sections based on token statistics, corresponding to the A-B evolution. The following table shows the averaged cosine distance between the respective section and Bio (our assumed endpoint of the A-B evolution).

Code:

Section  <cos(φ)> Herbal A    0.17 Pharma A    0.24 Astro       0.25 Cosmo       0.47 Herbal B    0.37 Recipes B   0.53 Bio B       0.74

With the exception of Cosmo, the cos(φ) values monotonically grow with the proposed section ranking. Is this merely by coincidence? As for Cosmo: while the cosine distance would suggest to rank this section following Herbal B (rather than preceding it), we put more weight on our previous token-based analysis here (because of the extremely low folio number).

See also the following table showing the frequencies for the most common token types for the Biological section (shedy, ol, chedy, qokedy, qokain, qokeedy, qol, qokal, shey, and chey):

Code:

          Herbal (A)  Pharma (A)  Cosmo+Astro  Herbal (B)    Stars (B)    Bio (B) shedy        (0.0%)    1 (0.0%)    17 (0.4%)    35 (1.1%)   113 (1.1%)   247 (3.6%) ol        58 (0.7%)   39 (1.5%)    43 (0.9%)    35 (1.1%)   111 (1.0%)   233 (3.4%) chedy      1 (0.0%)    1 (0.0%)    24 (0.5%)    62 (1.9%)   190 (1.8%)   210 (3.0%) qokedy       (0.0%)    1 (0.0%)     5 (0.1%)    39 (1.2%)    61 (0.6%)   164 (2.4%) qokain     1 (0.0%)    1 (0.0%)     6 (0.1%)     5 (0.2%)   105 (1.0%)   159 (2.3%) qokeedy      (0.0%)      (0.0%)     4 (0.1%)     9 (0.3%)   137 (1.3%)   153 (2.2%) qol        2 (0.0%)    4 (0.2%)       (0.0%)     1 (0.0%)    28 (0.3%)   116 (1.7%) qokal      2 (0.0%)    3 (0.1%)    16 (0.3%)     9 (0.3%)    41 (0.4%)   107 (1.5%) shey      40 (0.5%)   19 (0.8%)    23 (0.5%)     6 (0.2%)    84 (0.8%)    99 (1.4%) chey      55 (0.7%)   21 (0.8%)    22 (0.5%)    20 (0.6%)   123 (1.2%)    94 (1.4%)

We agree that the rigorous statistical significance of the direct folio comparison, and in particular of our figure 1, is difficult to assess. More advanced methods like PCA might gain better insight, but come with their own potential string of problems.