17-11-2025, 07:55 AM
(This post was last modified: 17-11-2025, 08:50 AM by Mark Knowles.)
(17-11-2025, 01:25 AM)dashstofsk wrote: You are not allowed to view links. Register or Login to view.But also there is a further problem. Something more serious for your hypothesis. If the top words are all filler words then once you have eliminated them the frequencies of the remaining words start to flatten. Among them there isn't any dominant common word, none that occurs significantly more than the next common word. In short, the non-filler words will not follow Zipf's law.
So then, since Zipf's law holds for most natural languages, and it won't under your hypothesis, where is your meaningful text?
I didn't say that "all the top words are filler words" that is your assumption. Clearly, amongst the real words there will be more common words like pronouns or conjunctions, so it wouldn't be as simple as saying the 80% most frequent words are filler words.
I would suspect that the longer of your "top words" are more likely to be filler words than the shorter of the most frequent words as the most common words in a language tend to be shorter words, so the most frequent real words would be short words. Of course, this does not mean that all short words are real words.

