Categorizing the text-only pages

[MarcoP]

I have been experimenting some more with page-similarity measures. I considered a few word-based options:

• lexicon-similarity: only considering word-types, the ratio between the intersection and the union of word-types in two pages
  
• word-histogram difference: Bhattacharyya difference between word histograms for the two pages (this is of course based on word tokens)
  
• token-overlap: the number of tokens that appear in both pages divided by the average number of tokens in the two pages
  

As a measure of "quality" I counted how frequently the verso of a page is found among the five best matches for the recto of the page. The idea is that there are good reasons to assume that "r" and "v" were written consecutively and they often appear to share similar illustrations, dialects and the same scribe. Of course there is no guarantee that this measure is meaningful: I used it just because it seems to make sense to me and it is simple to check.
According to this criterion, the three word-based methods perform better than the bigram-based method I previously posted. Among the three, token-overlap appears to be the best (but histogram is quite close and the lexicon method is not far either).

An example of how the method works:
PageA: one one two three
PageB: one one two two four five

The two pages share three tokens (one, one, two), the order is not significant. Average page-length is 5.
3/5=0.6
This measure falls in the 0-1 range, with 0 corresponding to no overlap between words and 1 to exactly the same word-tokens (possibly in a different order).

These are the measures I found for text-only pages (three best matches for each page, best match first):
1r 8v 0.266 101r 0.222 38v 0.215
57v 66r 0.193 49v 0.161 67r1 0.114
58r 58v 0.214 70r2 0.193 86v5 0.191
58v 107v 0.251 86v6 0.245 86v5 0.224
66r 81v 0.254 78r 0.230 75v 0.224
76r 116r 0.425 75r 0.395 103r 0.388
Ros 85r1 0.184 86v5 0.176 86v6 0.167
85r1 86v6 0.281 84r 0.265 78r 0.262
86v6 86v5 0.315 107v 0.290 85r1 0.281
86v5 86v6 0.315 86v3 0.270 107r 0.266
86v3 86v5 0.270 106v 0.259 106r 0.257

The similarity between 57v and 66r / 49v is largely due the lists of single-glyphs words.
Quire 14 appears to be quite coherent in terms of shared words (though the absolute values for Ros are low, maybe unsurprisingly since the diagram looks so "special").