About the generation of similar words

[quimqu]

quimqu

Can you you explain the communities a bit more.  It sounds a bit drastic for 38k tokens compared to the other Corpus's I'm guesing.  If the communities are the core logic of the language is that abnormal for a text of that size?

Hi,

I first build a graph where words are connected if their distance is ≤2 (less or equal at two changes in terms of characters). That gives a large connected component. Then I run a community detection algorithm on that graph.

So a community is just a dense cluster inside the Levenshtein ≤2 network. Words inside a community are very similar to each other, but not necessarily all within distance 1. Some can be at distance 2, or even further apart if they are connected through intermediate words. Note thatt distance is not Levenshtein; distance are the steps to cgo from a token to another).

I wouldn’t say communities are the “core logic” of the language. They are just clusters of words that are very similar in form. In natural languages, those clusters usually come from things like inflection, derivation, or spelling variation.

What is interesting in the Voynich is not that communities exist, but how large and dominant a few of them are. A big part of the vocabulary falls into just a few very dense clusters. So communities are more like a reflection of the system’s structure, not the underlying logic itself.