(I removed most of the quotes form previous posts, they were getting unwieldly)
I doubt the number of slots by itself is an useful metric, also because this number already enters in the calculation of Ngrammarsize (a series of multiplications extended to the number of slots) and eventually of Ncharset (a series of sums over the slots). And the degenerate "1*WS" grammar has just one slot, so it beats any other one on this count. But I'm not 100% certain, I didn't go deep enough in this.
Those are all good suggestions, thank you! But I need time to think about each, will see what comes out of it.
And of course I agree with you that, even if the 'evaluation problem' I posed is solved, it's yet not enough to say if a grammar is actually 'good' (and hopefully useful). But it's also true that's difficult to make progress if the basic questions are not answered first, and this was my aim: try to put the evaluation process on firmer basic mathematical grounds, then see if further progress can be made. This may not even be the best way to proceed, I don't know', I'm just trying what comes to my mind (and stands up to basic sanity checks
).
I also take the chance to compliment for your works on the Marginalia (just saw them yesterday, I have a lot of threads to read xD). I'd wish I had 1/100th of your skills and knowledge in paleography and image processing!! And of course, clues external to the bare transcribed text could be (and are) very, very useful.
(04-12-2024, 05:04 AM)oshfdk wrote: You are not allowed to view links. Register or Login to view.This is interesting. If the number of slots is irrelevant, then the trivial grammar comes at the top on each of three metrics: coverage (they all have 100%), efficiency and the number of elements. My intuition would be that the number of slots matters and my intuition would be that grammar [B|∅][AB|B|A][C|∅] is subjectively better than [BA|A|B][A|B|BC|C], because it includes two optional elements, effectively showing prefix-infix-suffix form (which may or may not correspond to some actual feature of the underlying language or could just be a statistical fluke), while grammar [BA|A|B][A|B|BC|C] appears to assign some chunks to prefixes and suffixes with no clear path for a language model. E.g., why BA is a prefix and BC is a suffix? Note that either grammar could be the true grammar of the original language, it's just that one of them attempts to extract functional elements from the text, and the other doesn't.
If we assume the number of slots is irrelevant, maybe the total combined length of all element strings is relevant? But in my opinion, all this is just guesswork, until the more fundamental problem below is addressed.
I doubt the number of slots by itself is an useful metric, also because this number already enters in the calculation of Ngrammarsize (a series of multiplications extended to the number of slots) and eventually of Ncharset (a series of sums over the slots). And the degenerate "1*WS" grammar has just one slot, so it beats any other one on this count. But I'm not 100% certain, I didn't go deep enough in this.
(04-12-2024, 05:04 AM)oshfdk wrote: You are not allowed to view links. Register or Login to view.The problem is if we are trying to optimize the quantitative (numeric) metrics of the grammars, we need a quantitative metric of the end task. We can try comparing the statistical or structural properties of the predicted and the actual sets. Something like (I'm just throwing in examples randomly): "median length of a grammar word divided by the median length of a text word", "median Levenshtein distance between a grammar word and the nearest text word", "total number of character bigrams predicted by the grammar but absent in the text", "same as the last one, but reformulated as probabilities: the probability of the actual count of a character bigram in the word list given the expected count of a character bigram in the grammar, assuming all the outputs of the grammar are equally likely", etc. There are many and many potential metrics of how well the predicted set matches the actual text, but without choosing a specific metric it's really hard to evaluate the grammars.
[Edit]: Maybe we even can reframe the metric as a linear algebra problem: given a slot grammar assign probabilities to each element (so that the probabilities sum to 100% for each slot) in such a way as to minimize the difference between the probabilities of the words of the predicted set (computed as the product of the probabilities of respective elements taken from each slot, I guess) vs the relative counts of the same words in the observed set (computed as the actual word count divided by the total number of tokens, 0 for missing words). This way we will try to mimic the actual text token distribution. Then compare the grammars using their respective optimal element probabilities with this metric (the difference between the probabilities of the words of the predicted set vs the relative counts of the same words in the observed set).
Those are all good suggestions, thank you! But I need time to think about each, will see what comes out of it.
And of course I agree with you that, even if the 'evaluation problem' I posed is solved, it's yet not enough to say if a grammar is actually 'good' (and hopefully useful). But it's also true that's difficult to make progress if the basic questions are not answered first, and this was my aim: try to put the evaluation process on firmer basic mathematical grounds, then see if further progress can be made. This may not even be the best way to proceed, I don't know', I'm just trying what comes to my mind (and stands up to basic sanity checks
).I also take the chance to compliment for your works on the Marginalia (just saw them yesterday, I have a lot of threads to read xD). I'd wish I had 1/100th of your skills and knowledge in paleography and image processing!! And of course, clues external to the bare transcribed text could be (and are) very, very useful.