Vord length distribution

[Alin_J]

Folio You are not allowed to view links. Register or Login to view. and the other dense-text folios have wide spaces and half-spaces. If the narrow spaces are ignored, you get 4- and 5-syllable tokens like pchedalshdy, yteechypchy, qotalshesy, qokeolkeedy, qokeykeedy, qokeeylkeol, tcheyqokeey, qokalshedy, pcholkchdy, oteeykshy, rotailshedy, qokeodair, oieshedy, chealainor, otaryly, oteeochey, qopchedy, ocheocthey, qoctheody, okeeolkeeodain, fchedykchedy, shoefcheeykechy (some of which do not have half-spaces, they are definitely one token). This is only a sampling. There are many more.

Another complication is that dalalchdar (debatably 3 or 4 syllables) is written with half-spaces, so is it one token or three?

Regarding treating certain or uncertain spaces as word-separators, for the 101 transcription if you treat them as real separators you get 9905 different word-types but not treating them as real spaces (concatenating "word parts") you get 10641 different word-types, a significant number more. I have therefore subsequently treated uncertain spaces as real separators. My argument is that if these spaces would have been random mistakes/seemingly spaces they would have likely lead to more different word-types when treating them as separators, not less, because they would shop off many legit word-types into new ones. Instead, concatenating real word-types randomly should lead to more word-types which is what is also observed for the text.