3 hours ago
(This post was last modified: 3 hours ago by npcompl33t.)
(4 hours ago)Jorge_Stolfi wrote: You are not allowed to view links. Register or Login to view.And comparing bigram statistics makes sense only if the spelling is mostly the same. If you take a text in German with umlauts and the standard spelling, and another text with ue/ae/oe and where "sch" is written "x" and "ie" is "i" and "ch" is "q" or "shh" depending on being hard or soft, you may well conclude that they are more different among themselves than either is to Basque or Armenian or whatever.
I suspect you know this is not really true.
But just in case, I happen to have reference texts for all of these, so I tested this statement using the exact same distance metrics I used to calculate my golf statistics. This is a great example, but not in the way you had implied.
Not only is the method sensitive enough to detect these differences accurately, it shows the difference within one of my subgroupings is around the same as the German spelling examples. While the differences between my subgroupings, spans the range of the difference between German and Basque.