An attempt at extracting grammar from vord order statistics.

[davidd]

I would have a question Davidd. Do you have any statistical simple measure that would tell us how "good" these groups are? Something like correlation coefficient ot some measure of total variance explained?

This definitely requires more statistical knowledge than I have, Rene or Nablator can certainly make better suggestions. My unreliable guess is that, as a simple measure of quality, one could compute the probability for a graph to generate the text under examination. For instance, one could use You are not allowed to view links. Register or Login to view.discussed in Brown et al. "Class-based n-gram models of natural language".



The probability of generating word Wi given that the preceding word is Wi-1, is the product of the probability of Wi occurring as a member of class Ci multiplied by the probability that class Ci-1 (Wi-1's class) is followed by class Ci (the arrows in Davidd’s graphs, assuming that the weights of arrows going out of a class add up to 100%).

One can compute such probabilities for all words W1...Wn and multiply all of them to get the overall probability for the whole passage.

I guess that Pr(Wi|Ci) is simply how many Wi tokens occur among all the tokens assigned to Ci.

This system appears to be simple enough, but I think it can only compare models based on the same number of classes and texts of identical length.

The standard deviation of a binomial distribution scales with the square root of the sample size
Imagine you have a 6 sided die (dice). How many casts (throw) do you have to do to test if it is a fair one? Mathematically you can never be sure it is totally fair, but you can put upperbounds on how unfair it is, what the maximum deviations of the odds are. If you throw it 60 times, you have less information than when you throw it 600 times. When statisticians say the expected value is 100, they off course dont mean that they expect litterally to have 100 throws of each value, but somewhere close to 100. 

My plan is to regard being in a group and transitioning to another group is like throwing a dice, and calculate how fair that dice is, how far those observations deviate from a "fair" dice assuming the odds of transition to a group would normally just depend on the relative group size that youre transitioning to.