Words get more unusual as they get longer (from 2 characters)
#3
The main "more unusual" trend is indeed found in all texts, including random ones.

For "overall average Levenshtein distance" I suppose that you calculated the average edit distance of all pairwise permutations of word types within each EVA length class.  I get values similar to You are not allowed to view links. Register or Login to view. using paragraph text in IT2a-n.txt (chosen for its alphabet-sized character set).  Each increment of word length adds approximately 0.8 edits to the "overall average":
   
We can convince ourselves (by thought experiment) that as the number of available characters increases, the plotted points will converge on the grey line of 1:1 slope.

Dividing the average edit distance by word length yields average edit distance on a per-character basis (the local slope):
   
Here the VMS transcription in gray is compared with "natural" texts of similar length:  early modern English (red, 1611 King James Gospels),  and medieval Latin (blue, Speculum Humanae Salvationis).  Open symbols represent a version of each text in which the character order has been pseudorandomly scrambled, preserving the word length and character frequency distributions.  The three scrambled variants are barely distinguishable in the plot.  Thus, the dominant contribution to edit-distance-unusualness is a combinatorial property unrelated to any fine structure of the vocabularies.

On closer inspection:
   
The "natural" text samples KJB and SHS show a small offset toward lower average edit distance from their scrambled counterparts, representing some kind of letter-order and -frequency correlations, not strongly dependent on word length.  Surprise! — Voynichese is the outlier.  Its longer "words" are progressively less mutually unusual (on a per character basis) than the shorter or scrambled ones.  Does it mean that longer words contain more longer-range correlations?  This topic may have been addressed on the forum previously, from a different perspective.

It has been a slow week in Analysisland.
[+] 4 users say Thank You to obelus for this post
Reply


Messages In This Thread
RE: Words get more unusual as they get longer (from 2 characters) - by obelus - 27-02-2025, 10:43 PM

Forum Jump:


Users browsing this thread: