Repetition of words

[Mark Knowles]

I have been thinking about a simple little algorithm which is to subtract all instances of words that are ever repeated in sequence in the manuscript. So for example taking the sentence:"I had a thought and I had had the opportunity to design this algorithm" then the word "had" is repeated and so all instances of this word should be removed from the text leaving "I a thought and I the opportunity to design this algorithm". This algorithm should be iterated recursively until there are no words that repeat in the text. It would be intriguing what the properties of the resultant text are and how much they differ from the original text. Quite a lot of words should be removed as if they repeat just once on just one page of the manuscript they should be deleted from the whole text of the manuscript.Words that only repeat once all other repeating words are deleted should also still be deleted.

Of course, I think that words that repeat have a very high probability of being filler words. There may be some filler words that don't ever repeat, but hopefully this process should help to prune the text considerably of many of the filler words.