Your concept of “trigger” words was not coming across very clearly before. That may be more my fault than yours, so let me see if I can get a few things straight.
As I understand it, you are treating a word that occurs anywhere on a page containing word doublets as a “trigger” if that word correlates highly with such pages. But the trigger word can appear anywhere on the page: before the doublet, after the doublet, between different doublets, and so on.
Also, there is no particular correlation between the trigger word and the specific words that form the doublet. So, for example, if “sy” appears on a page, it predicts that at least one doublet, or perhaps many doublets, are likely to occur on that same page. But it does not predict which particular doublet or doublets will appear. It could be a doublet of any word.
And if a word correlates negatively with pages containing doublets, then it is an “anti-trigger,” so to speak — what you call an inhibitor. In other words, it predicts the absence of doublets on the page where it occurs.
Regarding your cluster plot: you used the t-SNE algorithm to place the words as points in a three-dimensional space. Then you independently colored them according to whether they were positive predictors of doublets, which you call triggers, or negative predictors, which you call inhibitors.
Presumably, you did that by assigning any word with a coefficient greater than 0 as a trigger, and any word with a coefficient less than 0 as an inhibitor. Or did you use thresholds — for example, one color for words with coefficient greater than 0.5, and another for words with coefficient less than -0.5?
I also do not understand why “sy” appears while none of the other labelled words in that plot even show up in the bar chart of extreme correlation words. I assume the blue “sy” (Instead of red) is just a switch in color choices, but I still cannot understand the labelling on the cluster plot.
In any case, most of the individual points on your cluster plot are not visible, because they have largely been replaced by continuous contoured color shading. So it is not clear whether the intensity of that shading represents the value of the z-dimension or the density of points.
If the shading is based on the trigger/inhibitor assignment I described above, then with or without non-zero thresholds, what I am seeing is simply that the three reduced dimensions chosen by the t-SNE algorithm have some general relationship to the coincidence of a word with doublets occurring on the same page. But that relationship could be as simple as the page having or not having word doublets at all; it is not clear that the reduced dimensions are even capturing the existence of the trigger or inhibitor words. Also, regardless of whether the shading reflects coefficient value or point density, any clustering that is present in the plot seems to form three clusters about as strongly as it forms two.
In short, the connection between the results of the clustering plot and the coefficients that signal triggers and inhibitors, as illustrated in the bar chart, remains totally unspecified.
(02-07-2026, 02:56 AM)Vuk88 Wrote: You are not allowed to view links. Register or Login to view.As for the tests conducted, I can tell you that in each of the 1,000 permutations, the order of the words in the manuscript was completely randomized. However, I strictly preserved the exact global vocabulary, the frequencies of every single word, and the word lengths, while also standardizing the page lengths. I then ran the exact same predictive pipeline (Logistic Regression -> Accuracy Score) on the shuffled text just to see if the model could still predict the appearance of doublets based on the surrounding vocabulary. The predictive accuracy collapsed to about 50%, so pure randomness.
Can you clarify this further?
By “the same predictive pipeline,” do you mean that in each of the 1,000 trials, “sy” — and in fact
all the words — effectively became neutral triggers, with no real predictive power one way or the other?
Or do you mean that in each of the 1,000 trials, all the words were analyzed completely afresh to find a new ordering and range of logistic regression coefficients, and then the predictive accuracy was recomputed using those trial-specific trigger words? In that case, are you saying that no words showed meaningful predictive power, with all coefficients effectively close to zero? (And, of course, as I think someone pointed out earlier in the thread, you must keep the doublets together during the shuffling procdes, as if they were a single word.)
Obviously, under the first scenario, we should not be at all surprised that the “Actual Voynich Data” appears in the extreme tail of the distribution shown in the violin plot.
By the way, I did find that your Zenodo paper goes into a bit more detail than your initial post at the top of this thread, though not by very much.
I assume you have good answers to many of the questions I have raised here. If so, I think it would be well worth explaining them in your formal research paper. If the conclusions you have stated are correct, then you have a good story to tell.
But to be candid, I remain dubious. I still feel that your major finding is primarily what you have shown in your last “Cross Analysis” plot. Assuming that result is not eventually explained away by a coding bug, I think it could be analyzed much further than you have taken it so far.
For example: are the trigger words and inhibitors correlated with particular word doublets? That question alone could produce a different assignment of triggers versus inhibitors. Are the trigger words correlated with the number of word doublets on a page? What about the proximity of the trigger words to the predicted doublets within individual pages?
If I can find some time, I will try to reproduce your analysis.
Regards,
--Andrew
