30-04-2025, 07:35 AM
STEP 2
Until I get an answer about the total amount of tokens, I've constructed three structured random control datasets. These controls mimic specific properties of the EVA text while removing its internal structure and semantic content.
CONTROL2: Randomized Vocabulary
A new vocabulary is generated by creating random strings that match the exact lengths of the original EVA tokens. For example, if an EVA word has six characters, its corresponding CONTROL2 token will also be six characters, but composed of randomly chosen lowercase letters. Each TokenID from the original EVA vocabulary is mapped one-to-one to one of these artificial tokens. The resulting pages preserve the same page structure and token IDs per page, but with completely random token forms.
CONTROL3: Generic Token Pages
This control retains the same number of tokens per page as in the EVA dataset. However, instead of using any EVA vocabulary, it uses a generic pool of 2,000 artificial words (e.g., w1, w2, …, w2000). Tokens are randomly drawn from this pool with replacement, ensuring that the length and granularity of each page match EVA, but the vocabulary is entirely independent and non-semantic.
CONTROL4: Shuffled TokenIDs
This dataset preserves the overall frequency distribution of the EVA vocabulary. We extract all TokenIDs used in EVA and shuffle them. Then we reassign them to pages using the same token count per page as in EVA. In other words, the “words” remain the same, and their frequencies across the dataset are preserved, but their positions and associations across pages are randomized.
Why This Matters
These structured controls allow us to isolate which properties of the EVA text drive the observed structure. Each control preserves different features:
-CONTROL2 keeps token lengths but randomizes content
-CONTROL3 matches page lengths but uses unrelated vocabulary
-CONTROL4 preserves vocabulary and frequencies but randomizes order
Suppose the EVA dataset continues to show consistent and significant structure under these comparisons. In that case, it supports the hypothesis that the Voynich Manuscript contains a non-random, internally coherent system, possibly thematic or linguistic in nature.
If so, what?
Next Step
The next step will be to apply topic modeling (LDA) to each of these controls, using the same number of topics (4), which matches the prior analysis on EVA. By extracting latent topic distributions per page and comparing them to the EVA results, we will assess whether similar thematic structures emerge in the controls. If they do not, and the structure only appears in EVA, this would be strong evidence of internal organization unique to the original manuscript.
As always, I'll appreciate opinions!
Until I get an answer about the total amount of tokens, I've constructed three structured random control datasets. These controls mimic specific properties of the EVA text while removing its internal structure and semantic content.
CONTROL2: Randomized Vocabulary
A new vocabulary is generated by creating random strings that match the exact lengths of the original EVA tokens. For example, if an EVA word has six characters, its corresponding CONTROL2 token will also be six characters, but composed of randomly chosen lowercase letters. Each TokenID from the original EVA vocabulary is mapped one-to-one to one of these artificial tokens. The resulting pages preserve the same page structure and token IDs per page, but with completely random token forms.
CONTROL3: Generic Token Pages
This control retains the same number of tokens per page as in the EVA dataset. However, instead of using any EVA vocabulary, it uses a generic pool of 2,000 artificial words (e.g., w1, w2, …, w2000). Tokens are randomly drawn from this pool with replacement, ensuring that the length and granularity of each page match EVA, but the vocabulary is entirely independent and non-semantic.
CONTROL4: Shuffled TokenIDs
This dataset preserves the overall frequency distribution of the EVA vocabulary. We extract all TokenIDs used in EVA and shuffle them. Then we reassign them to pages using the same token count per page as in EVA. In other words, the “words” remain the same, and their frequencies across the dataset are preserved, but their positions and associations across pages are randomized.
Why This Matters
These structured controls allow us to isolate which properties of the EVA text drive the observed structure. Each control preserves different features:
-CONTROL2 keeps token lengths but randomizes content
-CONTROL3 matches page lengths but uses unrelated vocabulary
-CONTROL4 preserves vocabulary and frequencies but randomizes order
Suppose the EVA dataset continues to show consistent and significant structure under these comparisons. In that case, it supports the hypothesis that the Voynich Manuscript contains a non-random, internally coherent system, possibly thematic or linguistic in nature.
If so, what?
Next Step
The next step will be to apply topic modeling (LDA) to each of these controls, using the same number of topics (4), which matches the prior analysis on EVA. By extracting latent topic distributions per page and comparing them to the EVA results, we will assess whether similar thematic structures emerge in the controls. If they do not, and the structure only appears in EVA, this would be strong evidence of internal organization unique to the original manuscript.
As always, I'll appreciate opinions!