(30-06-2026, 10:58 PM)Vuk88 wrote: You are not allowed to view links. Register or Login to view.The Logistic Regression doesn't use an n-gram model, it uses a Bag-of-Words approach at the page level. So it's not looking at physical adjacency, but rather page-level co-occurrence. It found that words like shedy and chees heavily turn on or off on the exact same pages where the doubles appear.Thanks, it's clearer now.
(30-06-2026, 10:58 PM)Vuk88 wrote: You are not allowed to view links. Register or Login to view.On the sample size issue with the 36 chees vs 298 doubles, I had the exact same concern about statistical chance. That's precisely why I ran a Monte Carlo Permutation Test with 1000 iterations. The empirical p-value came out to p=0.0010, and the mathematical distance between the permuted random distribution and the true manuscript data is what confirms this isn't just a random fluke.
Question: did the MonteCarlo permutation reshuffle all the words, or did it keep the original doublets and reshuffled the rest? And, did it keep the page structure of the original (the number of words on each page)? I think I would be convinced your findings are not a statistical fluke with a permutation test which:
- keeps the same doublets as the original in the same positions (that is to say, on the same page as the original, given your algorithm works page by page)
- keeps the same page structure (number of words on each page) as the original
- avoids creating new doublets when reshuffling words
- and finally reshuffles words section by section (Balneo, Herbal A, Herbal B..). I say this because there are big differences in vocabulary between the sections and mixing everything together will tend by itself to dilute any statistical signal.
In any case, nice work
