Linguistic Patterns Before Decipherment: A Key to Understanding Unknown Texts

[Urtx13]

This feels reasonable to me (we’re removing only ~2.6% of the raw tokens), but maybe I’m missing something?

I agree 42852 tokens are too many. Hard to say from here what is not working properly.

Did you check if the [cth:oto] removal procedure recovers the correct tokens? You should get a 'cthres' from line 1
<f1r.1,@P0>      <%>fachys.ykal.ar.ataiin.shol.shory.[cth:oto]res.y.kor.sholdy<!doodle: @254;>

.. and an 'oteos' from line four:
<f1r.4,+P0>      soiin.oteey.oteo[s:r],roloty.cthiar,daiin.okaiin.or.okan


What does your processing do in cases where curly brackets are found, ie.
<f1r.17,+P0>      ycho.tchey.chekain.sheo,pshol.dydyd.cthy.dai[{cto}: @194;]y
<f1r.19,+P0>      dchar.shcthaiin.okaiir.chey.@192;chy.@130;tol.cthols.dlo{ct}o



Then, I'd check manually a few pages of the cleaned text vs. the original.

You chose a quite complicated transcription format for your work, why did you not use RF1a-n, which is simpler to manage? (just remove non-word characters and tokens including a '?'). The less time you spend on coding the text cleaner, the more time you'll have for your actual research.

Hola!

Thanks for your thoughts. You’re right that RF1a-n would have made things simpler for preprocessing. I agree from a software engineering perspective.

But in this project, I chose to work with ZL3a-n (EVA full) intentionally, not out of convenience, but because:
1. EVA is the community standard for most statistical and linguistic analyses of the Voynich Manuscript, especially in computational papers. Since I’m applying topic modeling, entropy, permutation tests, and other statistical tools, I wanted to remain compatible with existing literature.
2. I wanted complete control over the preprocessing pipeline so that I could experiment with variant handling, bracketed alternatives, special glyphs like @123;, and non-alphabetic tokens. Writing my parser (now IVTT-like) gave me insight into how different cleaning strategies affect downstream structure.
3. Reproducibility. One of my goals is to offer an end-to-end, reproducible pipeline from raw EVA to statistical validation. Having precise control over every transformation step, even the messy ones, is part of the methodology.
4. Personal challenge. Part of this is pedagogical—this is a research project with a strong exploratory dimension. I want to see how each decision (e.g., what to remove, what to retain) shifts the result.

Still, your advice is spot-on for production-quality workflows. I may eventually re-run the pipeline with RF1a-n or the IVTT output to compare token counts and structural consistency.

Thank you always for your kind answers!