Least effort generator
tikonen > 6 hours ago
Disclaimer. This is a hobby project and I'm not trying to solve the manuscript, I think it's a medieval scam. Just hoping to hear ideas, comments and critique.
My working assumption is that the manuscript is a forgery made to part rich people of their money. I've been thinking how it was generated in seemingly consistent way. Maybe it was done by sweat shopping few cleric students for pocket money and minimal time and and money was invested to train them.
So I guess there was possibly a template or a collection of tokens the scribes used to learn to generate VM's fantasy words with some personal twist, mistakes and tweaks. The binomial distribution of a word length gives some support to this idea, indicating that the word lengths may be random. After some practice the tokens were not needed anymore and scribe would just make it on the fly. VM's high word "kindness" to the previous rows could also support this idea.
To test the feasibility of this idea I wrote a simple program that when given a word from VM it finds a minimal number of predefined tokens in order to construct the word. I did not consider words that are used only twice or less and the ones with rare letters.
Here is and example of one token list set I conjured.
Word is constructed by picking a token from each list (or skipping it) and concatenating them to a word.
As you can guess list s0 is considered first when building a word, then list s1 and so on until s5 as last. A token from each list can be used only once for a word.
This limited example already covers (with above mentioned filtering) 34% of all the unique words and 75% of all words (many words are used multiple times). With this set the average number of needed tokens for common words with >=50 instances is 2.1. (For >=10 it's 2.5). So basically one can generate most VM words by picking just few tokens.
s0 = ['qok', 'qoke', 'qot', 'ot', 'd', 'q', 'l', 'yk']
s1 = ['ch', 'sh', 's', 'ok']
s2 = ['e', 'o', 'r', 'k']
s3 = ['r', 'k', 'o', 'e']
s4 = ['cth', 'ckh']
s5 = ['daiin', 'aiin', 'ain', 'ey', 'edy', 'dy', 'y', 'ar', 'or', 'al', 'am', 'air', 'ody', 'ol']
Example: Word 'chedy' would be token 'ch' from list s1 and 'edy' from s5.
Position on each token in each list reflects how often it's used in VM words. Each word may have multiple "solutions", algorithm picks shortest.
It's relatively easy to make a larger set for >95% word coverage but then of course the average number of token picks increases (slowly).
Some points:
- One can generate non VM words if tokens are chosen totally randomly. Some external rules are required.
- Real VM words do not use much s2 and s3. Omitting them drops coverage from 75% to 66%.
- Impossible to draw line how much is template and how much is just random on the fly variation of scribes
- There are some words that have high occurrence but complicate the set, they need a token that is used mostly on that word.
I don't know if this has any merit but at least it's an interesting exercise.