Right, I will have to check that later. Normally I'd prefer digits to be removed since more often than not they are part of modern layout.
I collected some numbers already, you can see them here (final sheet xMATTR):
You are not allowed to view links. Register or Login to view.
Of the texts I checked so far, here's the percentage that have no repeating strings of the following lengths:
2: 1%
3: 6%
4: 20%
5: 32%
6: 48%
So about half of these texts never repeat a string of 6 words exactly (taking into account the 1000-word window I always used, which seems reasonably large).
What may be more interesting is that 9 of my txt files never even repeat a 4-word string.
I have to leave now, but I'll already attach an example so someone could double check.
I collected some numbers already, you can see them here (final sheet xMATTR):
You are not allowed to view links. Register or Login to view.
Of the texts I checked so far, here's the percentage that have no repeating strings of the following lengths:
2: 1%
3: 6%
4: 20%
5: 32%
6: 48%
So about half of these texts never repeat a string of 6 words exactly (taking into account the 1000-word window I always used, which seems reasonably large).
What may be more interesting is that 9 of my txt files never even repeat a 4-word string.
I have to leave now, but I'll already attach an example so someone could double check.