About the generation of similar words

[evandrunen]

....
To test this, I stopped grouping tokens into “families” (that was too unstable) and instead looked only at direct pairs of similar tokens (Levenshtein = 1). Then I compared their contextual profiles. The result is quite strong: tokens that are formally similar have much more similar contexts than expected. Not slightly more. A lot more.
...

Could you explain what you mean by "context"? Sorry if I missed it.

[quimqu]

Could you explain what you mean by "context"? Sorry if I missed it.

Context means the environment in which a token appears. It is not about meaning, but about surroundings. It includes which tokens tend to appear nearby, how similar they are, how dense or repetitive the local area is, and also where the token sits in the structure, like its position in the line or the section of the manuscript.