Yesterday, 04:01 PM
Here is a basic question abut the methodology. Consider this simple example.
The text one each page of a document is a string of Nk decimal digits, 0 to 9. The digits of each page k are generated independently by a process that depends on a vector Pk of 10 parameters, where Pk[7] is simply the probability that the process spits a 7 at each step. The set of all possible Pk is the canonical 9-dimensional simplex S of R^10, with vertices (1,0,0.,.0), (0,1,0,..0), (0,0,0,..1).
Let Ck[7] be the number of times the digit 7 appears on page k. The JS distance is defined between two probability distributions, so we need to estimate the probabilities Pk from the counts Ck. The naive way is to use the raw freauencies, Pk[7] = Ck[7]/Nk. Then of Ck[7] is zero, we get Pk[7] = 0.00; f Ck[7] = 1 and Nk = 100, we set Pk[7] = 1/Nk = 0.01, etc.
A better way is to use Bayes, assuming that any vector Pk in the simplex S is equally likely. I forgot what is the formula, but if Ck[7] = 0 and Nk is 100, it gives something like Pk[7] = 0.005, and if Ck[7] = 1, it gives something like Pk[7] = 0.015.
Is this how you compute the statistics vector of each page? (Except of course that you count 256 digraphs or other things).
All the best, --stolfi
The text one each page of a document is a string of Nk decimal digits, 0 to 9. The digits of each page k are generated independently by a process that depends on a vector Pk of 10 parameters, where Pk[7] is simply the probability that the process spits a 7 at each step. The set of all possible Pk is the canonical 9-dimensional simplex S of R^10, with vertices (1,0,0.,.0), (0,1,0,..0), (0,0,0,..1).
Let Ck[7] be the number of times the digit 7 appears on page k. The JS distance is defined between two probability distributions, so we need to estimate the probabilities Pk from the counts Ck. The naive way is to use the raw freauencies, Pk[7] = Ck[7]/Nk. Then of Ck[7] is zero, we get Pk[7] = 0.00; f Ck[7] = 1 and Nk = 100, we set Pk[7] = 1/Nk = 0.01, etc.
A better way is to use Bayes, assuming that any vector Pk in the simplex S is equally likely. I forgot what is the formula, but if Ck[7] = 0 and Nk is 100, it gives something like Pk[7] = 0.005, and if Ck[7] = 1, it gives something like Pk[7] = 0.015.
Is this how you compute the statistics vector of each page? (Except of course that you count 256 digraphs or other things).
All the best, --stolfi