About the generation of similar words

[quimqu]

I extended the analysis a bit further, still keeping the same constraints: no parents, no direction, no predefined structure.

The question I wanted to answer next was simple:if tokens are locally similar, do they also behave similarly in context?

To test this, I stopped grouping tokens into “families” (that was too unstable) and instead looked only at direct pairs of similar tokens (Levenshtein = 1). Then I compared their contextual profiles. The result is quite strong: tokens that are formally similar have much more similar contexts than expected. Not slightly more. A lot more.

Group	Mean cosine	Median	Q25	Q75
Formal pairs (lev = 1)	0.6917	0.7003	0.5975	0.7716
Matched random controls	0.4664	0.4688	0.3489	0.5577

The average similarity between real similar pairs is around 0.69, while matched random pairs (same length and frequency constraints) are around 0.46. And this survives a permutation test very easily.

So this adds something important to the previous post: local similarity is not just geometric. It is also functional. Form and context are clearly coupled.

At the same time, the “cloud vs chain” picture still holds. Even though similar tokens share context, that similarity is not organized around a single dominant neighbor. It is still distributed across the local neighborhood.

I also revisited the sequential side more carefully (using interpolated bigrams and proper train/test splits). There is a small sequential signal, but it is weak, and it is not stronger than what you get after shuffling within pages. So sequence alone does not seem to explain much of the structure.

I also tried a simple HMM to see if a few latent states could explain the patterns. That did not work well at all. The model collapses and does not generalize to test data (it assigns zero probability in several cases), and the inferred states are not clean or interpretable. So at least in this form, a small-state discrete model does not seem to capture the structure.

Summarizing, we could say:

• There is strong local clustering of similar forms.
  
• This clustering is symmetric in time (reading direction).
  
• Similar forms share similar contexts (with a strong signal).
  
• The structure is not dominated by nearest-neighbor copying.
  
• Simple sequential models are weak.
  
• Simple latent-state models fail.
  

So the picture that emerges is still the same, but now more constrained: it really looks like a local space of related forms, where tokens are selected from a structured neighborhood, rather than generated through a linear derivation process.

This is also where your comment fits in well, @Torsten. Your heatmaps already showed that similarity increases symmetrically around a token, not just from the left. What I think these additional results add is that this symmetric field is not only spatial, but also contextual, and that it does not reduce to a single dominant predecessor even locally.

The open question, at least for me, is now this: what kind of process generates something that behaves like a structured local cloud of related forms, rather than a sequence where each token clearly comes from the previous one?