qo- vord formation

[RenegadeHealer]

Hi RenegadeHealer ,
as stated, the contest would find that this script is unbeatable (100% accurate):

for i in range(0,38000):
    print "daiin"

 Where shall I wire the funds?

Maybe you mean that we should compare word frequencies in the output with actual word frequencies in the manuscript (so that daiin should occur about 850 times, each of ol, chedy, aiin about 500 and so on).

Indeed, that makes much more sense. The standard of perfection that this kind of contest would be aiming for would be an output that computerized textual analysis had low confidence in distinguishing from the original VMS.

Something very similar can be done with the grammar that Stolfi built 20 years ago (see You are not allowed to view links. Register or Login to view.): differently from most grammars (e.g. what Thomas posted at the start of this thread) Stolfi includes numerical weights for each rule. So, while in Thomas' model 'k' and 'f' are totally equivalent, Stolfi also models the fact that 'k' is about 30 times more frequent than 'f' (second number in each row):

G:
5858 0.34755 0.34755 t
1243 0.07375 0.42130 p

9423 0.55906 0.98036 k
 331 0.01964 1.00000 f


There is no doubt that Stolfi's model (however good) can be improved, but is getting a better fit for word frequencies the most promising task on which we should spend our money (or time)?

I'm not really sure, to be honest. What makes me think it could be a promising avenue is some reading I've been doing recently about tracking — identifying animals (or people, if you're a detective) by their footprints. You can tell a lot about an animal's physiology just by studying the pattern of its footprints. Similarly, Voynichese is a pattern of imprints. We might be able to tell a lot about the kind of system that produces that kind of pattern of imprints, the better we understand the subtle complexities of the pattern.

For example, given the differential frequencies of [k] and [t], I know that the text probably wasn't composed by randomly rolling a die that had an equal number of faces with [k] options and [t] options. But a die that had about twice as many [k] options than [t] options isn't ruled out. This is kind of like how when I see tracks that involve two deep and wide impints behind two shallow and closely spaced imprints, I know it could be a rabbit, because it would take an animal with back legs far stronger than its front legs to make a track like this.

I read Stolfi a long time ago, really should reread his paper. What it left me wondering was, what kind of information lends itself to being notated with a tri-part system like his core, mantle, and crust, each with the number of degrees of freedom that each of these word parts has? I'm intrigued, for example, with Linda Snider's idea that each vord could be a geographical or mapping coordinate. I would have to know much more about how people in Medieval Eurasia thought about and conceptualized geography to evaluate the merits of that idea, but it's a good example of how I'm thinking about how one could reconstruct the system that outputted the VMS's text, just by the properties of the text alone.

Another participant to your contest could be Timm and Schinner's algorithm (see You are not allowed to view links. Register or Login to view.r). Like Stolfi's model, their algorithm contains several numerical parameters and one could tweak them to get a better fit for word frequencies. But they have chosen to follow a different line, investigating other properties of the text, rather than focusing on word structure. For instance their algorithm reproduces these phenomena:

• the progressive drift in word frequencies through the text (what was initially seen as two different "languages" Currier A and B);
  
• reduplication and quasi-reduplication (words repeating consecutively, identically or with minimal changes);
  
• line-effects: words at the beginning or end of lines are different from other words.
  

Though I don't think that Timm and Schinner come closer to actual word frequencies than Stolfi, their work marks a significant step forward, building on Stolfi's grammar by integrating word structure with other parts of the larger picture.

Absolutely. I think these are the top contenders so far. And Timm and Schinner's paper shed a good bit of light on the type of system that could potentially produce an output like the VMS. I seem to remember Stolfi's did likewise, though if I remember correctly, the patterns he found led him to different conclusions about the kind of system that could account for those patterns.

Another recent "generative system" that adds to the field, without addressing the area of word frequencies is You are not allowed to view links. Register or Login to view..

Personally, I would not be terribly interested in a complex software (say, a You are not allowed to view links. Register or Login to view.) that produces a perfect word histogram and tells us nothing about dialects/language-drift, reduplication, first-last combinations (the influence of the last character of a word on the first character of the following word), the relationship between labelese and paragraph text, etc. Not only I believe that all features should be explained together (and Timm and Schinner have done the most extensive work in this direction) but I am sure there are many more features and patterns that have not been discovered yet (see Lisa Fagin Davis' ongoing research).

I do think the simplest and fairest way to do a contest like this would be to award the most perfect histogram. But I think a lot of the fun would be looking at each of the entrants one by one, and asking ourselves, "What would it take to use this tool to produce an output that exhibited, for example, the drift from one Currier language to another?" And so on for any of the unique features that continue to be elucidated in the VMS. If there is no simple way to imagine tweaking one of these tools to produce something that has the subtler features of the VMS text, then it's probably not an accurate reconstruction.