05-02-2018, 06:20 PM
Rene - I don't have those articles you cite. Are they available?
Anton - rather than suggesting infinity, Heap's suggests that the dictionary size continues to increase as the collection adds documents, and that no maximum dictionary size is ever reached. Which is logical, as in natural texts the entire corpus of the OED will never be used - but the more text there is, the more likelyhood there is of rare and technical terms being used. We see a massive increase at the beginning of the text which quickly trails off.
As for Heap's observation with different natural languages:
Torstein - you are correct I only used one point for each Currier language. It is entirely possible that the first few pages use the bulk of the dictionary and then repeat those words.
Anton - rather than suggesting infinity, Heap's suggests that the dictionary size continues to increase as the collection adds documents, and that no maximum dictionary size is ever reached. Which is logical, as in natural texts the entire corpus of the OED will never be used - but the more text there is, the more likelyhood there is of rare and technical terms being used. We see a massive increase at the beginning of the text which quickly trails off.
As for Heap's observation with different natural languages:
Quote:Zipf's law on word frequency and Heaps' law on the growth of distinct words are observed in Indo-European language family, but it does not hold for languages like Chinese, Japanese and Korean. These languages consist of characters, and are of very limited dictionary sizes. Extensive experiments show that: (i) The character frequency distribution follows a power law with exponent close to one, at which the corresponding Zipf's exponent diverges. Indeed, the character frequency decays exponentially in the Zipf's plot. (ii) The number of distinct characters grows with the text length in three stages: It grows linearly in the beginning, then turns to a logarithmical form, and eventually saturates.
Linyann Lü et al, You are not allowed to view links. Register or Login to view.
Torstein - you are correct I only used one point for each Currier language. It is entirely possible that the first few pages use the bulk of the dictionary and then repeat those words.