Ran some new tests on my generated code vs the Voynich.
This test measures novelty, predictability, word beginnings, word middles, word endings, and how strongly words cluster into families. It's from some code I developed months ago. It basically trains on half the text then tests on the other half to see how they compare.
In this test the generated code is mostly in the ballpark with the exception of bigram reject. That's basically a measure of, look at the bigrams on one half of the text and then see which of those bigrams are in the other half. In the case of my generator, it's pretty low which means, it's producing too many close relatives. It stays inside the learned families extremely well, but the real herbal text appears to generate more new combinations while still obeying its structural constraints.
BPC is bits per character which measures entropy or how predictable text is. BPC2 uses bigrams, BPC3 trigrams. ΔBPC is the difference between the two.
The ED1 tests look at how many words in the training set are edit distance 1 or copies of words in the test set. Token asks how many words have a close relative in the test set and type asks how many unique words have a close relative. Again, it's off by a bit and is probably related to the bigram reject difference. 97% of the unique training words are ED 1 or 0 to a test word.
And I think I know why these are issues. I tried to keep everything a copy or ED1 from it's source and it almost never generates a random new word. It appears there's a bit more novelty in the Voynich than I'm allowing for.
So again, not perfect. In the neighborhood? Mostly, but needs work.
This test measures novelty, predictability, word beginnings, word middles, word endings, and how strongly words cluster into families. It's from some code I developed months ago. It basically trains on half the text then tests on the other half to see how they compare.
In this test the generated code is mostly in the ballpark with the exception of bigram reject. That's basically a measure of, look at the bigrams on one half of the text and then see which of those bigrams are in the other half. In the case of my generator, it's pretty low which means, it's producing too many close relatives. It stays inside the learned families extremely well, but the real herbal text appears to generate more new combinations while still obeying its structural constraints.
BPC is bits per character which measures entropy or how predictable text is. BPC2 uses bigrams, BPC3 trigrams. ΔBPC is the difference between the two.
The ED1 tests look at how many words in the training set are edit distance 1 or copies of words in the test set. Token asks how many words have a close relative in the test set and type asks how many unique words have a close relative. Again, it's off by a bit and is probably related to the bigram reject difference. 97% of the unique training words are ED 1 or 0 to a test word.
And I think I know why these are issues. I tried to keep everything a copy or ED1 from it's source and it almost never generates a random new word. It appears there's a bit more novelty in the Voynich than I'm allowing for.
So again, not perfect. In the neighborhood? Mostly, but needs work.
| Metric | Herbal Voynich | RK42 | Difference |
|---|---|---|---|
| Bigram reject % | 0.5655 | 0.1044 | much lower |
| BPC2 | 2.4472 | 2.8717 | higher |
| BPC3 | 2.1978 | 2.5697 | higher |
| ΔBPC | 0.2494 | 0.3020 | higher |
| Initial BPC | 3.0692 | 3.1479 | slightly higher |
| Medial BPC | 3.3306 | 3.4473 | slightly higher |
| Final BPC | 2.6014 | 2.9747 | higher |
| ED≤1 Token | 95.7988% | 98.9295% | higher |
| ED≤1 Type | 88.0416% | 96.9503% | much higher |