About the construction of lines in the MS

[nablator]

Removing spaces does not kill the effect, and keeping them improves the model, which suggests that the spacing, even if imperfect, still reflects some real structure in the text.

Spaces are mostly "correct" relative to the obvious word structure, but sometimes when several choices seem plausible for space placement (for example gallows and y can be either the first or the last character) the scribes seemed to have been ignorant or careless, either appending the character to the word on the left, or prefixing it to the word on the right, or leaving it floating between words, as if they had no idea where it should be, or putting a small space to signify their indecision, or skipping the 2 spaces entirely. In these cases transliterations are all over the place, randomly ignoring some visible spaces and adding some invisible spaces... So it seems to me that at the heart of the word structure mystery is a hidden "correct" tokenization that scribes were only approximating and didn't know how (or care) to enforce strictly.

Note: the Zattera 12-slot sequence is ambiguous for the purpose of re-spacing space-less Voynichese, unless a greedy pattern matching is done, but the result would be different if the matching is done right-to-left instead of left-to-right, this is what I mean exactly by "ambiguous". Which unambiguous sequence is the "correct" one and why (if such a thing has a usefulness for defining the actual building blocks of Voynichese) is a difficult question, some sequences might be optimal for some metric... Perhaps some compression algorithm could suggest an optimal choice of tokens, as has been suggested recently, but every algorithm's choice of text chunks for its dictionary is different. Perhaps a better way to approach the problem would be to find the best unambiguous slot sequence (used in a simple regular expression for greedily matching the space-less text to generate chunks) by the hill climbing algorithm, using Shannon's entropy to calculate the theoretic compression factor, then check if there is a universal slot sequence, generally optimal or if several locally optimal sequences exist for different sections of the VMS. This is something I wanted to try You are not allowed to view links. Register or Login to view. but was unsure how to proceed. I think I'll do it this week-end.

Sorry for hijacking your thread for this off-topic discussion.