11-05-2018, 10:19 PM
(09-05-2018, 03:15 PM)MarcoP wrote: You are not allowed to view links. Register or Login to view.I think it could be interesting to perform more structured experiments along these lines, adding more quantitative indexes that could help measure distance between languages. Also, there are other corpora that one could try with this or similar approaches.
Both the two lowest-entropy languages (Vai and Korean) are not written in the Latin alphabet. Both alphabets appear to be syllabic.
Korean is alphabetic, though it might not look like it. The letters are grouped into syllables, each of which is a Unicode character (3 UTF-8 bytes). For example, the word Samsung is 삼성, the syllables are 삼 (sam) and s성 (seong), and the letters are ㅅ (s),ㅏ(a), ㅁ (m),ㅓ (eo), and O (ng).
If you want to process Korean script as alphabetic, you need to transliterate the syllables into letters first.