01-10-2017, 02:04 PM
(01-10-2017, 01:46 PM)Koen Gh. wrote: You are not allowed to view links. Register or Login to view.Wouldn't this also mean that the algorithm will perform poorly in languages which prefer large consonant clusters?
The algorithm tries to predict the next character on the basis of the single preceding character.
It the input is made of clusters of letters that tend to occur consecutively (e.g. clusters of consonants followed by clusters of vowels) the model will favor sequences of characters from the same state and the A matrix will have high probability loops
state:0->state:0 state:1->state:1
and lower probability
state:0->state:1 state:1->state:0
If there is alternation (as it typically is the case with consonants and vowels) it will favor state alternation with the probability of state:0 being followed by state:1 being higher than state:0 being followed by state:0 (low probability loops).
I guess the algorithm will perform poorly in languages having frequent trigraphs, since these cannot be model by only considering one character and the immediately following one. Anyway, at the moment, I don't know how to compare results of runs based on different inputs.

