Writing a fake Voynich by rules: copies, habits and the link between words

[quimqu]

Hi, I try to answer all of you:

Why 6? Should it be 8 (length of without_)?

good catch, and thank you @nablator !. It’s a bug that appeared when I translated the code from Catalan to English, sorry. (the option was “sense_M” (6 letters) and became “without_M” (8), but the code kept cutting 6, so it excluded nothing, ups). The "blind" quire 13 I regenerated yesterday had in fact seen quire 13... It’s fixed now and the blind file regenerated: words with e are 35.9% in the blind skribe, against 50.6% in the real quire and 49.8% in the skribe trained on everything (with the bug it was 47.2%).

I also added what you asked: SCRIBE_TRAIN=with_M (or with_A+B+C) trains only on those quires, lang_A or lang_B only on one Currier language, and sec_herbal (or other sections) only on those sections. Run calibrate_section_e.py and calibrate_length.py with the same SCRIBE_TRAIN first; they use the same filter, which is why they looked like they use everything. quire.py writes each case to its own file.

About mixing A and B: the glyph tables are not mixed, and this is important. Each hand has its own table, smoothed only with the table of its own language, and each hand and section has its own on top of that. What is shared are some global settings (how often it copies, how strong the habits are, part of the line-end closing, and the word list it uses to know which words are rare). With lang_A or lang_B even that is separate. And every page is written on the layout of the same real page (same hand, section, lines and words per line), so you can compare page by page.

How does it actually work?

@Mauro sorry for the confusion, both things are true. The skribe writes glyph by glyph, each glyph depending on the two before it (and the first one on how the previous word ended), with tables learned for each hand and section. The template isn’t used to build the word, it’s a check: if the word that comes out doesn’t fit the order prefix, core, e, ending, it’s thrown away and drawn again, and words with two gallows or with e and i together are mostly rejected too. So the template describes what comes out, but the writing is glyph by glyph.

The copying only uses words already written on the same page. I find this important: a page starts empty, and the first line of a paragraph copies very little, so the first words are all new. After that about one word in five is copied from the page, mostly from the last few lines, with a bonus for the word just above; in almost half of the copies one glyph is changed, with the same two-glyph memory. It resets at every page, not at every bifolium. What the bifolium shares are the habits (k or t, ch or sh, q, dy or y, words with e), not the words.

Samples by section: quire 13 (biological) is in scribe/quire_M, real, trained on everything and blind. With the new option the skribe can also be trained on one section only, so I can post samples like that if it helps. And all the generated text has the ZL3b line identifiers, so anyone can run npcompl33t’s seriation on it.

I double-checked if the fix changed anything else. It only affected the without_ option, so the normal skribe should write exactly the same text, and it does: I generated the whole manuscript with the old code and with the new one and compared them, and they’re identical. I also ran the full battery again (5 runs, 82 measures) and the odd/even test, and every measure gives the same value as before:

Comparison	Before the fix	After the fix
Skribe vs real text	78/82 (median error 0.060)	78/82 (median error 0.060)
Skribe learned from odd folios, writing the even ones	69/82 (0.068)	69/82 (0.068)
Real odd vs real even folios (the ceiling)	73/82 (0.049)	73/82 (0.049)
Three-rule generator (baseline)	44/82 (0.254)	44/82 (0.254)

Big thanks to you, guys!