About the generation of similar words

[quimqu]

The results I shared earlier were not reliable, so I removed them (sorry) and restarted the analysis from scratch with a stricter and more agnostic approach.

The main issue was methodological. I was implicitly assuming that similar tokens have a "parent" in the recent past, and then analyzing the data under that assumption. That can easily create patterns that look real but are actually imposed by the pipeline.

This time I avoided that completely. No parents, no bursts, no direction assumed. What I did instead was simple:

• For each token, I measured how many similar tokens (Levenshtein ≤1 and ≤2) appear nearby.
  
• I compared past vs future symmetrically.
  
• I checked whether similarity is concentrated around one specific neighbor or spread across many.
  
• I validated everything against multiple null models.
  

Two things are clearly true:

• There is strong local clustering of similar forms. Tokens are not placed randomly. They tend to appear in regions where many similar tokens exist.
  
• This clustering is symmetric in time. Similar tokens are just as likely to appear after a token as before it.
  

I then looked at how that local similarity is structured.

   

This plot shows how much of the local similarity is explained by the single best match. If a “parent-like” mechanism dominated, we would expect values close to 1. Instead, most of the mass is well below that. The best individual match typically explains only a small fraction of the local similarity. In other words, tokens do not seem to be anchored to a single dominant neighbor.

   

This makes it even clearer. As the local field becomes denser (more similar tokens nearby), the importance of the best individual match systematically decreases. This is a very strong pattern:

• when there is only one similar token, it trivially dominates,
  
• but as soon as multiple similar tokens appear, the structure becomes distributed,
  
• the system behaves more like a “cloud” than a chain.
  

First conclusion: The data does not support a simple "copy from a previous token and modify it" mechanism. This is solid because:

• it does not rely on defining parents or bursts,
  
• it holds across different similarity thresholds and window sizes,
  
• it survives null model comparisons,
  
• and it is confirmed by multiple independent signals: symmetry in time, absence of directional bias, and lack of a dominant local match.
  

What this does NOT rule out? The absence of a simple directional signal does not mean there is no structure or no generation process. The data is still compatible with:

• reversible or cyclic transformations,
  
• selection within a local repertoire of similar forms,
  
• systems with latent states activating families of tokens,
  
• or other non-linear, non-directional mechanisms.
  

So, the structure looks more like a local space of related forms than a linear chain of derivations.

I also compared two explicit models on held-out pages: a local cloud model, where a token is supported by the density of similar forms around it, and a copy-modify model, where it is supported by specific similar tokens in the past. Both outperform a unigram baseline, but the local cloud model wins consistently across the entire parameter grid. This does not prove that the text is generated non-directionally, but it does show that local distributed similarity explains the data better than a simple retrospective copy-modify mechanism.

I will keep posting the next steps as I go, focusing on what is actually supported by the data and separating it from hypotheses. Sorry for my previous deleted post.