Writing a fake Voynich by rules: copies, habits and the link between words
#11
(Yesterday, 03:44 PM)npcompl33t wrote: You are not allowed to view links. Register or Login to view.Do you have any controls with known semantic meaning? What would happen for example if you applied your model to half of the pages of Moby-Dick. Wouldn't it also produce english-like text that passes many of the same checks?

Hello, I haven't tried this so far... as said, I just took "the scriba" as a tool for checking rules. It is a good idea and I might do this. Thank you.
Reply
#12
(Yesterday, 03:18 PM)quimqu wrote: You are not allowed to view links. Register or Login to view.
(Yesterday, 01:58 PM)Mauro wrote: You are not allowed to view links. Register or Login to view.Hello @quimqu, it's always a pleasure to hear from you, interesting food for thoughts as always.

Among the battery of tests, did you include the standard ones (entropies, distributions...). I also remember you had developed some methods which explored the long range structure of text (I can't find the reference now, I remember you compared the VMS to many different texts). Did you try that too?

Some of them, yes, and I just ran the rest on the scribe's text (real vs scribe, average of 5 runs):

Character entropy h1: 3.88 vs 3.86
Conditional entropy h2: 2.12 vs 2.09, h3: 1.85 vs 1.84
Zipf slope: 1.00 vs 1.00

Word length distribution is close but not right: the scribe makes too many words of 2-3 glyphs and too few of 5-6.

Long range is where it fails. In the real text, neighbouring pages share more vocabulary (0.35 cosine, falling to 0.17 at 100 pages apart) and frequent words come in bursts. The scribe is too uniform (0.29 falling only to 0.19), because it only copies within the page. So there is some drift across pages that my rules don't capture yet. Good catch!

Quick update on the long range thing you asked about, Mauro.

I looked at where the similarity between nearby pages comes from, and it's not a drift in time. If I make the scribe copy from the previous pages it gets worse, the whole curve goes flat. What realy matters is the hand, and the section inside the hand. In the real text, pages of the same hand and section are more alike than in my scribe, and pages of different hands are less alike.

So I changed one thing: the scribe now learns its letter habits seperately for each hand, and for each section within a hand, instead of one table for language A and one for B. With that the curve comes out almost right:

page similarity at distance 1 / 10 / 100
real: 0.35 / 0.26 / 0.17
before: 0.29 / 0.24 / 0.19
now: 0.33 / 0.26 / 0.19

Bursts of frequent words go from 1.51 to 1.59 (real 1.76) and MATTR from 0.60 to 0.62 (real 0.64). It also holds when the scribe learns only from the odd folios and writes the even ones.

Two side effects I didn't expect. Once every hand has its own vocabulary, the scribe needs much less copying from the same page (22% instead of 35%), so part of what looked like copying was just the hand's own words. And I also fixed the word lengths, it was making too many short words, now it follows the real distribution by position in the line.

What is still missing: pages far apart are still a bit too similar (0.19 against 0.17), so there is some diference between parts of the book that hands and sections don't explain. Could be a slow drift, or just that the Davis hands and the sections are too coarse as labels.

Thanks for remarking this, it was a good question.
Reply
#13
I will try now to explain in plain words what are the rules that the scribas follow (in my gnerator,of course).

What all the scribes do

The glyphs. About twenty signs. Some pairs seem to be the same sign used in different places: k and t, ch and sh, p and f. Wich one you get depends on the place and on the habit of the moment, not on the word. q always goes before o and only at the start of a word. e, i, n and m never start a word, and e, a, i, ch and q never end one.

The word. Always built with the same template: maybe a start (qo, o, d, s or y), then a core that is a gallows (k t p f) or a bench (ch sh), sometimes one or two e, and an ending. There are two families of words and they don't mix: words with e end in y or dy (qokedy, chedy), words without e end in aiin, ar or ol (qokaiin, dar, chol). Never more than one gallows. Most words have 4 to 6 glyphs. To write the next glyph you only need to remember the two before, and the endings work more like a way of closing the word than like real letters.

Between words. How a word ends decides how the next one starts. After a word in y, a word that would start with o usually gets a q in front (shedy okedy becomes shedy qokedy), and a word that would start with a gets a d (aiin becomes daiin), and after r or s you put nothing. What matters is the last stroke of the pen, not the word. Apart from that contact no word tells you anything about the next one, so there are no sentences. Even where the spaces go can be guessed from the glyphs around them.

The line. The line is the unit of work. The first word starts with d, s or y, or directly with a gallows, and never with a. Its first glyph avoids the first glyph of the line above (but not the one two lines above). The second word is the shortest. The last word gets "closed": aiin becomes am, ol becomes oly, ar becomes ary, y becomes g. This happens because it's the last position, not because the space runs out, it closes the same if it reaches the margin or not. The last line of a paragraph closes less.

The paragraph. It starts with a word with a gallows, almost never with q. The first line is special: it uses p and f instead of k and t (even more if there is free space above), the words are longer and newer, and nothing is copied.

Drawings. The text seems to be written after the drawings. When a line runs into a drawing the scribe treats it like a margin: he half closes before it and starts again after it like it was a new line.

Page and sheet. The scribe copies words he can see on the same page, mostly from a few lines up, often changing one glyph. Now I know this is at most one word in five, the rest of the repeats come from each hand's own vocabulary. Sometimes a word is repeated right after itself, mostly chol and qokedy, never daiin. Every sheet (the folded piece of vellum, four pages) has its own preferences, shared by the recto, the verso and the other half of the sheet: k or t, ch or sh, q or not, dy or y, more words with e or without, and even the size of the glyphs. It looks like a writing session.

Labels. The labels next to the drawings have their own form: they start with o, have no q, and end in y, ly or ry (otaly instead of otal).

None of these rules is about meaning. They are all about shape, place and process.

How each scribe is different

The five hands follow the same rules. What changes is how strongly they apply them, and also the vocabulary: each hand has its own set of words, and it shifts a bit from one section to another.

Hand 1, herbal A (112 pages, You are not allowed to view links. Register or Login to view. to f102v, also the pharmaceutical, about 9,700 words). Short lines (7 words) and short words (4.4 glyphs). Mostly the family without e, only one word in four has an e: daiin, chol, chor, shol, dain, dar. When it does an e word it prefers y over dy. Much more ch than sh, and less q than the others. It likes the gallows with a bench: cthor, cthy, cthol, otchol, kchor. It closes the last word only a little (aiin becomes am about a third of the time) and it's the one that avoids most strongly the first glyph of the line above. The link between one word and the next is the weakest (0.12 bits against 0.22 for hand 2).

Hand 2, biological (46 pages, You are not allowed to view links. Register or Login to view. to f86v, also part of herbal B and some text only pages, about 10,000 words). Lines of 8 or 9 words, words of 4.8 glyphs. Half of the words have e and they end more in dy than in y: shedy, chedy, qokedy, qokeedy, plus qokain and qokal. Lots of q and more sh than hand 1. Its own tic is starting with ol before a gallows: olkedy, olkain, olky, olkar, qol. It closes more than hand 1, and it has the strongest link between words.

Hand 3, the stars, the recipes section (31 pages, You are not allowed to view links. Register or Login to view. to f116r, about 10,800 words). The longest lines (10 words) and the longest words (5.1 glyphs). Mixes both families: chedy, qokeey, qokeedy, qokaiin, okaiin, aiin. Its tic is l before k: lkeey, lkaiin, lkeedy, lkain, and also qokair and otair. It's the one that closes most, at the end of a line aiin becomes am almost always. It's also the one that repeats a word right after itself the most, and it uses less p and f in the first line.

Hand 4, astronomical, zodiac and rosettes (10 pages but only about 500 words of running text). Almost everything it writes are labels. From the little text there is: short lines, words like daiin, ar, aiin, dal, dam, shey, and little q. Not enough text to say much more.

Hand 5, a few pages of herbal B and text only (7 pages, You are not allowed to view links. Register or Login to view. to f66v, about 800 words). The one with most e words (55%), mostly in dy. The one that changes k and t to p and f most in the first line. It closes a lot too, and at the same time it avoids the gliph of the line above the least. The only one that sometimes starts a line with a. Also little text.

One caveat. The five hands are Lisa Fagin Davis attribution, and Torsten Timm thinks its one scribe whose hand changed over time. With the text alone I can't decide that. Hands 1, 2 and 3 have enough text for the profiles to be solid, 4 and 5 not so much. And each hand almost matches one section, so part of what I call "a scribe" could also be "a section".
Reply
#14
Quote:I looked at where the similarity between nearby pages comes from, and it's not a drift in time. If I make the scribe copy from the previous pages it gets worse, the whole curve goes flat. What realy matters is the hand, and the section inside the hand. In the real text, pages of the same hand and section are more alike than in my scribe, and pages of different hands are less alike.



Interesting, I would be incredibly curious what happens if you were to attempt to generate the entire Herbal A hand one section, and measure accuracy, my guess would be it would be poor, because there is substantial variation in the words that are presented in this section, and even the frequency of common words like 'chol'. 

I suspect this is probably related to the "differences between parts of the book that hands and sections don't explain".

I would be curious what happens if you tried at the bifolio order, using You are not allowed to view links. Register or Login to view.I suspect your error will be heavily concentrated towards the end/start of the herbal A sequence, and be best performing in the middle. Or are you saying that the accuracy just degrades the longer it is run?
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Evidence for 2 rules linking words in Herbal A->B npcompl33t 30 3,951 08-10-2026, 05:37 AM
Last Post: npcompl33t
  Could synthetic “fake Voynich” tests tell us what we’re dealing with? Phoenixjn 4 4,277 10-09-2025, 09:19 AM
Last Post: dexdex

Forum Jump:


Users browsing this thread: 1 Guest(s)