06-02-2018, 05:31 AM
(This post was last modified: 06-02-2018, 05:40 AM by DonaldFisk.)
(05-02-2018, 06:20 PM)davidjackson wrote: You are not allowed to view links. Register or Login to view.As for Heap's observation with different natural languages:
Quote:Zipf's law on word frequency and Heaps' law on the growth of distinct words are observed in Indo-European language family, but it does not hold for languages like Chinese, Japanese and Korean. These languages consist of characters, and are of very limited dictionary sizes. Extensive experiments show that: (i) The character frequency distribution follows a power law with exponent close to one, at which the corresponding Zipf's exponent diverges. Indeed, the character frequency decays exponentially in the Zipf's plot. (ii) The number of distinct characters grows with the text length in three stages: It grows linearly in the beginning, then turns to a logarithmical form, and eventually saturates.
Linyann Lü et al, You are not allowed to view links. Register or Login to view.
I skimmed through the paper and it appears that they're using hanzi when processing Chinese text, kanji or kana for Japanese, and syllables for Korean, when they should be using words (which are generally polysyllabic in all three languages, and in Korean are separated by spaces). The number of words in each of the three languages will be similar to European languages and I would be surprised if Zipf's Law and Heaps' Law don't apply to them.

