Yesterday, 04:31 PM
(This post was last modified: 11 hours ago by npcompl33t.)
The Emerging Question
The question here ultimately is going to be " is it possible to connect words from 1a all the way through to Bio in a way that preserves information and frequency". Or more plainly is an evolution sequence incompatible with preserving semantic meaning.
This is ultimately the debate between semantic meaning and self-citation. The evolution sequence or gradient alone does not mean there can be no semantic meaning. The question is whether the transformations destroy information or whether they preserve it.
I'm testing how much information survives between words at the start of the network, all the way through to the end, using the same golf sequence clusters (1a, 1b, 1.5a etc) as nodes along the path. It needs to tracks both frequency mass and information.
The goal is to see:
Simple First Method
I made a quick simple model to simulate this process, that tracks information merges across each step like
This would be something comparable to:
So in the pronoun example, information is lost, but it keeps track of how many words get blurred together. If he / she still appears a few times outside of the merge, it may be that information is still preserved in contextually relevant situations.
I am tracking both overall frequency mismatches, as well as information preservation, across several different linkage methods between sections.
Initial Tests
In my initial tests, allowing merges based on edit distances destroys information, exactly as you would expect under Torsten's theory, while also allowing for almost perfect frequency preservation. This isn't unexpected, it means that its easy to find a path through the chain using generic edit distance links that preserves frequencies, at the cost of destroying most of the information.
My first simple path optimizer was able to find some paths that preserve 94% of word information, but can't seem to achieve better than 50% final usage mismatch.
This results for example with chedy containing the following ( chol 49.9%, chor 44.0%, cheol 4.1%, cheor 2.0% · <0.1% other values).
Vulgate Comparisons
However, when i test Vulgate using similar metrics, i get an average usage mismatch of 58.8%, a common word frequency mismatch of 24.5%, while still preserving 95% of information. Basically, vulgate combines words with different frequency profiles into a single token, causing a high frequency mismatch, while ultimately still preserving 95% of total information. This puts it roughly in the range of my initial simple voynich tests, close enough to justify more testing.
The New Bar
The bar is set.
The goal is now to try to determine if a 'path' can beat the null / edit distance model. Any solutions or mappings will need to prove they can work across the entire evolution sequence.
We should try to see if we can locate the path that appears to make the most sense, evaluate if it preserves meaning and keeps frequency mismatch within acceptable ranges. I'm working on building out a more complex path optimizer, You are not allowed to view links. Register or Login to view..
Ideally we need to also do some more work on historical vulgate controls to help refine the range of values that may be acceptable.
The question here ultimately is going to be " is it possible to connect words from 1a all the way through to Bio in a way that preserves information and frequency". Or more plainly is an evolution sequence incompatible with preserving semantic meaning.
This is ultimately the debate between semantic meaning and self-citation. The evolution sequence or gradient alone does not mean there can be no semantic meaning. The question is whether the transformations destroy information or whether they preserve it.
I'm testing how much information survives between words at the start of the network, all the way through to the end, using the same golf sequence clusters (1a, 1b, 1.5a etc) as nodes along the path. It needs to tracks both frequency mass and information.
The goal is to see:
- How much of the vocabulary can survive the overall transformation chain, and how much pure information is conserved
- what 'preserved' information looks like concretely, in terms of merged forms
- how important the specific connection model is to this preservation, whether different models result in drastically different rates
Simple First Method
I made a quick simple model to simulate this process, that tracks information merges across each step like
- if chol -> chedy and chor-> chedy then chedy = (50% chol, 50%chor)
- if chol + shol + qol = ol , ol = (33% chol, 33% shol, 33% qol)
This would be something comparable to:
- he + she + they = it , it = (33%he, 33%she, 33%they)
So in the pronoun example, information is lost, but it keeps track of how many words get blurred together. If he / she still appears a few times outside of the merge, it may be that information is still preserved in contextually relevant situations.
I am tracking both overall frequency mismatches, as well as information preservation, across several different linkage methods between sections.
Initial Tests
In my initial tests, allowing merges based on edit distances destroys information, exactly as you would expect under Torsten's theory, while also allowing for almost perfect frequency preservation. This isn't unexpected, it means that its easy to find a path through the chain using generic edit distance links that preserves frequencies, at the cost of destroying most of the information.
- 1 edit distance transformations results in 65% information retention, with a final usage mismatch of only 17% (0% = perfect frequency match)
- 2 edit distance has only a 3% usage mismatch but information preservation drops to 49% (all words) and (45% common words)
- 3 edit distance has a 1.5% usag mismatch, only 44% of all word information and 37% of common word information is retained
My first simple path optimizer was able to find some paths that preserve 94% of word information, but can't seem to achieve better than 50% final usage mismatch.
This results for example with chedy containing the following ( chol 49.9%, chor 44.0%, cheol 4.1%, cheor 2.0% · <0.1% other values).
Vulgate Comparisons
However, when i test Vulgate using similar metrics, i get an average usage mismatch of 58.8%, a common word frequency mismatch of 24.5%, while still preserving 95% of information. Basically, vulgate combines words with different frequency profiles into a single token, causing a high frequency mismatch, while ultimately still preserving 95% of total information. This puts it roughly in the range of my initial simple voynich tests, close enough to justify more testing.
The New Bar
The bar is set.
- A simplified version of Torsten's model allowing edit-distance based links preserves frequency mass almost perfectly, while destroying over half of information in the process. Note, this does not actually completely rule out semantic meaning. In this case, it would be extremely degenerative, but about 37-50% of information is preserved, depending on how relaxed the rules are. This still could be some type of degenerative or poor abbreviation system, or perhaps later LFD hands did not actually understand Voynich and did not effectively reproduce LFD hand 1's system
- It is possible to preserve information at the cost of frequency matching. Initial tests suggest this can still roughly fit within the expected values of an abbreviation system like vulgate.
The goal is now to try to determine if a 'path' can beat the null / edit distance model. Any solutions or mappings will need to prove they can work across the entire evolution sequence.
We should try to see if we can locate the path that appears to make the most sense, evaluate if it preserves meaning and keeps frequency mismatch within acceptable ranges. I'm working on building out a more complex path optimizer, You are not allowed to view links. Register or Login to view..
Ideally we need to also do some more work on historical vulgate controls to help refine the range of values that may be acceptable.