- Options
- Login
- Desktop Style
Options
A One-Page Ledger Method for Generating Voynich-Like Text
- A One-Page Ledger Method for Generating Voynich-Like Text
-
RE: A One-Page Ledger Method for Generating Voynich-Like Text
bi3mw > 59 minutes ago
Here's a very quick preliminary test. In my opinion, the result is acceptable for such a short text.
Code:#Word Types: 2177
#Word Tokens: 9502
#Words top 120
1 123 daiin
2 96 daiinal
3 63 qokainal
4 61 qokeey
5 54 okaiin
6 50 cheol
7 50 chey
8 50 chor
9 49 chol
10 47 chdy
11 47 chhy
12 47 okaiinol
13 42 cheor
14 42 okol
15 39 qokedyol
16 39 sheody
17 38 chedy
18 37 sheedy
19 36 chinal
20 36 qokeedyol
21 35 chear
22 33 cheey
23 32 cheydy
24 32 sheey
25 31 dair
26 31 okedy
27 31 otal
28 31 shdy
29 30 dain
30 29 chdyol
31 29 oteedy
32 29 sheol
33 28 otaiinol
34 28 sheor
35 27 olor
36 27 oteey
37 27 oteeydy
38 27 qokeeydy
39 27 shol
40 26 aiin
41 26 chain
42 26 cheeydy
43 26 okeey
44 26 otey
45 26 qokeydy
46 26 qoky
47 25 chedyol
48 25 choiin
49 25 okeedy
50 24 choldy
51 23 aiiin
52 23 chal
53 23 choky
54 23 okar
55 23 olol
56 23 shedy
57 23 shey
58 22 aiir
59 22 cholky
60 22 okeol
61 22 olky
62 21 chety
63 21 okody
64 21 shdyol
65 21 sheoraiin
66 21 shor
67 21 ykeol
68 20 aiinal
69 20 cheo
70 20 dady
71 20 okeal
72 20 oteal
73 20 qokal
74 20 ytchy
75 20 ytor
76 19 chty
77 19 okey
78 19 oldy
79 19 otar
80 19 otchy
81 19 ykol
82 18 cheedy
83 18 cheeey
84 18 cheoar
85 18 cheoor
86 18 choraiin
87 18 okady
88 18 olar
89 18 olchy
90 18 otear
91 18 oteeey
92 18 oteody
93 18 qoin
94 18 shal
95 18 ykaiin
96 18 ytaiin
97 18 yteol
98 17 chchy
99 17 okor
100 17 otain
101 17 oteol
102 17 oteydy
103 17 qochy
104 17 qokdyol
105 17 qokedy
106 17 qoteo
107 17 sheydy
108 17 shody
109 17 ykeedy
110 16 chir
111 16 choty
112 16 okair
113 16 okeeydy
114 16 olkydy
115 16 otaiin
116 16 qoal
117 16 qokaiin
118 16 qokeeody
119 16 qokor
120 16 qoty -
RE: A One-Page Ledger Method for Generating Voynich-Like Text
Dunsel > 35 minutes ago
(59 minutes ago)bi3mw Wrote: You are not allowed to view links. Register or Login to view.Here's a very quick preliminary test. In my opinion, the result is acceptable for such a short text.
And you see what I was saying about word length. You have no 1, 2 or 3 length words so you distribution is way to the right. Voynich is word length 5 centered, not 6.
Look at this output from my generator:
Output tokens: 8274
Output vocabulary size: 2525
Output hapax count: 1551
Voynich comparison tokens: 8519
Voynich vocabulary size: 2436
Voynich hapax count: 1642
So, your vocabulary is in the ballpark, maybe a little light. You add those length 1 and 2 tokens and that'll bring up that vocabulary count and likely bring your word length closer to the Voynich if you keep the token count the same.
Now, look at my top 20 bigrams and trigrams compared to the Voynich. It still needs some tweaking but it's in the ball park. Is yours?
Code:Top 20 output bigrams (100 pages):
ho: 2062
ch: 1689
sh: 1551
ai: 1319
ol: 1241
he: 1196
ey: 1037
in: 989
da: 884
ii: 851
ar: 761
or: 626
th: 621
ct: 611
od: 579
hy: 568
ha: 513
cp: 451
ph: 421
ot: 363
Top 20 Voynich bigrams (100 pages):
ch: 3082
ho: 1920
ai: 1433
in: 1374
ii: 1281
ol: 1239
he: 1116
da: 1116
sh: 1033
dy: 1024
hy: 948
ok: 930
or: 922
qo: 764
ot: 749
ar: 562
ey: 554
od: 548
ee: 541
eo: 467
Top 20 output trigrams (100 pages):
sho: 850
aii: 736
hey: 734
cho: 729
hol: 706
iin: 638
dai: 537
cth: 529
cph: 388
she: 362
che: 361
hod: 307
ckh: 259
oda: 252
ain: 249
hoy: 243
dar: 190
cfh: 188
chy: 187
har: 184
Top 20 Voynich trigrams (100 pages):
cho: 1236
iin: 1181
aii: 1139
dai: 728
che: 718
hol: 596
chy: 492
sho: 467
hor: 467
cth: 435
tch: 420
kch: 392
edy: 366
qok: 359
she: 295
hey: 291
ody: 253
qot: 251
otc: 244
heo: 241
Look at my word length compared to the Voynich then compare it to yours.
Look at my zipf curve compared to the Voynich. Does yours compare? I'm betting that until you fix the word length it'll be a "no".
These are the the very basic numbers a generator needs to be "plausible" with before you can even call it a generator. AND, to be a workable generator it has to be mostly blind about it. I do use a seed page. After that, it generates the rest on it's own just using my ledger. And all the numbers you see mine generate are emergent. I didn't create any code to force a zipf curve. I do have code that manages word length and that's something a scribe could do with their eyeballs.
The big thing you have to think about. If you sat down and tried to create something that looked like the Voynich and YOU were in the 15th century, what "tools" would you need other than a quill, some ink and maybe some ponce to erase things with. Any generator we come up with today needs to be human 15th century doable. Which is why the Naibbe cypher looks great and would technically be human doable but would require dice and cards and lookup tables... extremely cumbersome. Those are the type things you need to consider.
Creating Voynich-like text is easy.
Creating a method that explains how a 15th century scribe could do it, that's the hard part.
Edit: If you want to really see how you're doing, put up some real Voynich comparison charts, word length for word length. -
RE: A One-Page Ledger Method for Generating Voynich-Like Text
Dunsel > 10 minutes ago
Ran some new tests on my generated code vs the Voynich.
This test measures novelty, predictability, word beginnings, word middles, word endings, and how strongly words cluster into families. It's from some code I developed months ago. It basically trains on half the text then tests on the other half to see how they compare.
In this test the generated code is mostly in the ballpark with the exception of bigram reject. That's basically a measure of, look at the bigrams on one half of the text and then see which of those bigrams are in the other half. In the case of my generator, it's pretty low which means, it's producing too many close relatives. It stays inside the learned families extremely well, but the real herbal text appears to generate more new combinations while still obeying its structural constraints.
BPC is bits per character which measures entropy or how predictable text is. BPC2 uses bigrams, BPC3 trigrams. ΔBPC is the difference between the two.
The ED1 tests look at how many words in the training set are edit distance 1 or copies of words in the test set. Token asks how many words have a close relative in the test set and type asks how many unique words have a close relative. Again, it's off by a bit and is probably related to the bigram reject difference. 97% of the unique training words are ED 1 or 0 to a test word.
And I think I know why these are issues. I tried to keep everything a copy or ED1 from it's source and it almost never generates a random new word. It appears there's a bit more novelty in the Voynich than I'm allowing for.
So again, not perfect. In the neighborhood? Mostly, but needs work.
Metric Herbal Voynich RK42 Difference Bigram reject % 0.5655 0.1044 much lower BPC2 2.4472 2.8717 higher BPC3 2.1978 2.5697 higher ΔBPC 0.2494 0.3020 higher Initial BPC 3.0692 3.1479 slightly higher Medial BPC 3.3306 3.4473 slightly higher Final BPC 2.6014 2.9747 higher ED≤1 Token 95.7988% 98.9295% higher ED≤1 Type 88.0416% 96.9503% much higher