Categorizing the text-only pages

[MarcoP]

Hi RobGea,
I looked for similar pages using  an approach similar to what was discussed You are not allowed to view links. Register or Login to view. (but with a few differences).

• I extracted page text from the Zandbergen-Landini transliteration file ZL_ivtff_1r.txt, ignoring uncertain spaces. I considered all text (paragraphs, labels, anything). EVA was converted to CUVA.
  
• For each page, I computed the % of occurrences for 7 CUVA bigrams that are known to be indicative of different dialects: SO, DY, OL, ED, OR, HO, EO (EVA:cho, dy, ol, ed, or, qo, eo).
  
• I used Euclidean distance between the 7-dimensional vectors for each page to find the closest match to each text-only page.
  

As always, it is possible I made errors in the process. Take everything with a grain of salt.
These are the results  (text-only page, best match, distance):
1r 42v 2.74259639758
57v 73r 2.4187029582
58r 89v1 2.44073534002
58v 44r 3.31252622631
66r 105r 1.86491393903
76r 82r 2.28714013563
Ros 105v 2.5224805252
85r1 112r 1.81561862736
86v6 107v 2.05433857969
86v5 55r 1.12671469326
86v3 94v 0.966269113653


Obviously, some pages do not contain much text, so the results are not guaranteed to always be statistically relevant. In particular 44r, 73r and 94v have less than 100 words each.

The attached plots illustrate the results. I simply used PCA to reduce the number of dimensions to 2, this has the effect that the closest match often is not the closest dot in the plots. Green dots are the text-only pages and orange dots the best matches. I could not think of a simple way to connect the couples, but I think the plots are readable. The second plot zooms on the central area where all the relevant dots appear.