01-09-2019, 01:08 PM
The simplest method is probably just to eliminate the 10-20 most common words.
It's very much dependent on exactly what it is that you want from the data.
Take folio.f65r with 3 words only, 'otaim dam alam'.
otaim occurs in 1 other folio -> f111v[recipes].
dam occurs in 67 other folios ->Occurs in all sections.
alam occurs in 7 other flolios -> 5 times in recipes[f104r, f107r, f108r, f111r, f111v], 2 in text only[f58r,f58v]
shared words:
(2, 'f58v', {'dam', 'alam'}), (2, 'f111r', {'dam', 'alam'}) (2, 'f104r', {'dam', 'alam'})
(2, 'f111v', {'otaim', 'alam'})
Which of these results is the most useful ?
It's very much dependent on exactly what it is that you want from the data.
Take folio.f65r with 3 words only, 'otaim dam alam'.
otaim occurs in 1 other folio -> f111v[recipes].
dam occurs in 67 other folios ->Occurs in all sections.
alam occurs in 7 other flolios -> 5 times in recipes[f104r, f107r, f108r, f111r, f111v], 2 in text only[f58r,f58v]
shared words:
(2, 'f58v', {'dam', 'alam'}), (2, 'f111r', {'dam', 'alam'}) (2, 'f104r', {'dam', 'alam'})
(2, 'f111v', {'otaim', 'alam'})
Which of these results is the most useful ?