03-07-2019, 04:15 PM
(03-07-2019, 03:16 PM)Koen G Wrote: You are not allowed to view links. Register or Login to view.
I collected some numbers already, you can see them here (final sheet xMATTR):
You are not allowed to view links. Register or Login to view.
Of the texts I checked so far, here's the percentage that have no repeating strings of the following lengths:
2: 1%
3: 6%
4: 20%
5: 32%
6: 48%
So about half of these texts never repeat a string of 6 words exactly (taking into account the 1000-word window I always used, which seems reasonably large).
What may be more interesting is that 9 of my txt files never even repeat a 4-word string.
Additionally, one file (attached) does apparently not even repeat 3-word strings. It's only 1762 words, which might explain a bit.
Still, I would seriously question the claim that natural languages cannot be without repeating "phrases". If the phrase is 4 words long, I have 9 counterexamples. Make the phrase 5 words long and 20% of checked texts don't repeat any.