About the generation of similar words

[Torsten]

The problem with "bursts" detection by edit distance ≤ 2 is that it adds a lot of false positives to the group, unrelated words that happen to be "similar", especially short words like "or" and "dy" that are as dissimilar as they can be.

With such an unreliable definition I'm afraid that the dependency reconstruction is mostly illusory.

The criticism about false positives with raw Levenshtein distance ≤ 2 is valid, especially for short words. In my own implementation I addressed this by using a weighted edit distance that takes into account which glyphs the scribe would plausibly substitute for each other. Substituting similar glyphs (e.g. o/a/y, d/l/s, k/t/p/f) costs 1, while substituting dissimilar glyphs costs 2. This reduces false positives because two short words like "or" and "dy" score a high distance even though their raw Levenshtein distance is low.

The implementation is available at:
You are not allowed to view links. Register or Login to view.