GPT and the Voynich. Code! Not a solution!

[Dunsel]

Voynich words peak sharply at 5–6 glyphs, with a long but low-frequency tail.

That "long-frequency tail" is probably transcription errors, where multiple words are strung together.  Hunt them down and check with the page images.

You seem to be computing the length distribution of tokens (word ocurrences).  Try instead that of lexemes ("word types").

Be sure to map everything to lowercase.

For a double surprise, try also Chinese or Vietnamese...

All the best, --stolfi

Thanks.  Right now I'm just trying to make sure I can get accurate output from ChatGPT to run in Colab.  I know some of those errors like chokeeokychokoran in 49v exist where Takahashi jammed 2 words together.  I also know you fixed a lot of them in your Interlinear but be damned if I can get GPT to build a solid parser for it.  I'm going to give it another try tomorrow.

I've heard your theories that it may possibly be a phonetic east Asian language.  I was wondering today if the 'chunking' and syllabic–polygraphic system may have also been applied to an Asian language.  From what I saw of the charts today, longer word languages seem to fit into that syllabic–polygraphic cypher method that was used moreso than shorter languages like French.

Once I'm sure I can get reliable code from GPT, and so far it's doing exactly that, I'll start a new post that starts at the beginning of just getting the basic charts done and see where that leads.

Thank you so much for the suggestions and encouragement! I will work on them!