22-06-2025, 05:34 AM
(21-06-2025, 10:07 PM)Mauro wrote: You are not allowed to view links. Register or Login to view.Considering only the 'normal' (=~ more frequent) words, instead, largely avoids this problem, but there is no obvious break in the frequency data to suggest where the cutoff should be set, resulting as a minimum in a high degree of arbitrariness (and of course, if @Mark Knowles's hypothesis above is the right one, studying the normal words would be a waste of time).Studying the normal words would only be useful in so far as it helps to better identify what are normal "filler" words and what are real words. I would guess that around 20% of words are real words whilst the remaining 80% of words are just filler. This is just a guess, however I would be surprised if less than 10% are real words or alternatively more than 30% are real words.
It could be that the real text is encoded with a very simple substitution cipher in which case frequency analysis etc. should make the cipher easy to break. It could be that the real text is encoded with a complex cipher in which it will.be very hard to break especially in the context of having already to distinguish it from filler text.