Jorge_Stolfi > Yesterday, 10:11 AM
(Yesterday, 08:06 AM)Aga Tentakulus Wrote: You are not allowed to view links. Register or Login to view....
Aga Tentakulus > Yesterday, 10:50 AM
tavie > Yesterday, 01:13 PM
JoeyB > Yesterday, 05:29 PM
Vuk88 > 1 hour ago
(Yesterday, 05:29 PM)JoeyB Wrote: You are not allowed to view links. Register or Login to view.Question about the monte carlo sims, in your 03 stat valid python I see how you're defining any page with a doublet as 'infected' and then have the script take the whole page, remove the duplicate word (so, qokedy qokedy chol daiin -> chol daiin), and carve the page up into 30 word chunks. Is this right? Did you mean for the script to leave one "qokedy" in that example or is it intended to remove both?
Then you take every 30 word chunk and you say if this came from a page that contained a doublet somehwere on that page, model should label it infected. If page didn't contain a doublet, the model should label it clean.
Then your monte carlo (the permutation_test) shuffles the labels and the classifier runs again. But it isn't shuffling voynich words, it's shuffling labels. So you're testing: do 30 word chunks from doublet pages have a different vocabulary profile from 30 word chunks coming from non-doublet pages better than a shuffled label baseline. And I guess, if the model is allowed to see the doublet word as part of the vocabulary profile of the 30 word chunk is tied to whether you're removing one or both instances of th doublet word?