(10-02-2023, 04:03 PM)MarcoP wrote: You are not allowed to view links. Register or Login to view.I am aware that perfect reduplication is a special case of a more general phenomenon.I would like to use a metric that works when word boundaries are unknown, because they often are, if they are not completely fictitious (a possibility).
Could the (1) local diversity and (2) dissemblance between chunks of text be measured by the compression ratio of zip (or better algorithm)? (For comparison between different pages/sections only.)
(1) compressed_size(chunk)/uncompressed_size(chunk)
(2) compressed_size(chunk1+chunk2)/(compressed_size(chunk1)+compressed_size(chunk2))

