My knowledge of statistics is what it is, so I couldn’t fully understand the code screenshot at You are not allowed to view links. Register or Login to view. and the 3 lines of text at post #67.
I understand that class transitions are compared against the standard deviation of a random scenario (words randomly scrambled or random-word classes, I am not sure?). The distance “d” from this random scenario is expressed as a multiple of std dev and we would like its absolute value to be high: it’s OK both to have class transitions that are clearly avoided, resulting in low negative values (e.g. “the of”, for Genesis) and class transitions that are clearly preferred, with high positive values (“the father”).
A major difference of Voynichese (e.g. QuireM=Q13) with respect to the Genesis is that for English the diagonal has consistently negative values (words of the same class rarely appear consecutively), while for Voynichese there is no clear pattern: there are negative values with classes that avoid occurring consecutively (e.g. “shedy” and “ol”), positive values for classes that tend to repeat (“qokedy” and “okedy”) and three classes with values close to 0, i.e. behaving like the random case.
I guess this can be described as “autocorrelation”. Gaskell and Bowern (You are not allowed to view links. Register or Login to view.) discuss something similar about the distribution of word lengths:
The publication they mention is Matlach et al. You are not allowed to view links. Register or Login to view. (which I don’t think I ever entirely read).
It would be interesting to see if the languages that show word-length autocorrelation also show some measure of word-class autocorrelation (but I am not sure in which languages Gaskell and Bowern observed that feature).
Another observation is that, once one has a score function, it is possible to tackle the problem as optimization (e.g. by simulated annealing): this could possibly lead to better results, though it might be computationally complex and it’s possible that the improvement is only marginal.
I understand that class transitions are compared against the standard deviation of a random scenario (words randomly scrambled or random-word classes, I am not sure?). The distance “d” from this random scenario is expressed as a multiple of std dev and we would like its absolute value to be high: it’s OK both to have class transitions that are clearly avoided, resulting in low negative values (e.g. “the of”, for Genesis) and class transitions that are clearly preferred, with high positive values (“the father”).
A major difference of Voynichese (e.g. QuireM=Q13) with respect to the Genesis is that for English the diagonal has consistently negative values (words of the same class rarely appear consecutively), while for Voynichese there is no clear pattern: there are negative values with classes that avoid occurring consecutively (e.g. “shedy” and “ol”), positive values for classes that tend to repeat (“qokedy” and “okedy”) and three classes with values close to 0, i.e. behaving like the random case.
I guess this can be described as “autocorrelation”. Gaskell and Bowern (You are not allowed to view links. Register or Login to view.) discuss something similar about the distribution of word lengths:
Quote:A notable feature of the VMS that has to our knowledge only been discussed by one other publication [20] is positive autocorrelation of word lengths. Word lengths in most meaningful texts are negatively autocorrelated: that is, long words tend to be interspersed with short words (long-short-long-short). By contrast, the VMS exhibits positive autocorrelation (long-long-short-short). Positive autocorrelation is only observed in a limited number of natural languages, but is common in gibberish (Figure 3).
The publication they mention is Matlach et al. You are not allowed to view links. Register or Login to view. (which I don’t think I ever entirely read).
It would be interesting to see if the languages that show word-length autocorrelation also show some measure of word-class autocorrelation (but I am not sure in which languages Gaskell and Bowern observed that feature).
Another observation is that, once one has a score function, it is possible to tackle the problem as optimization (e.g. by simulated annealing): this could possibly lead to better results, though it might be computationally complex and it’s possible that the improvement is only marginal.