Mapping Voynich connections through rare tokens

[quimqu]

So out of all the rare words ( of which I count 1963 ) 84 fall exclusively within the Herbal pages? Also 82 ( Herbal-Stars ) fall exclusively within both the Herbal and Stars pages? Similarly, 5 exclusively within the Text pages? Or have I got this wrong?

But also what is your objective? Is it to determine whether rare words are distributed randomly or are localised within sections or by illustration type?

It seems to me that localised is going to be adequately explained under both the meaningful and meaningless hypotheses.

After the null normalization most Herbal connections basically disappear. That does not mean Herbal has no shared tokens. It means the amount of overlap is close to what we would statistically expect simply because Herbal is such a large section with lots of text and lots of opportunities for random overlap.

So the graph edges are not “exclusive words between two sections”. An edge just means that pages from those two sections share a certain amount of semi-rare tokens. Some of those same tokens may also appear in other sections as well.

What becomes interesting is when a connection is much stronger than the null expectation. That is what happens mainly with Marginal stars and some Text-only folios. Those sections still keep unusually strong links after normalization, while Herbal mostly falls back to expected levels.

And yes, my original motivation was basically this: hapax legomena are too isolated to build relationships, while the very common tokens are almost everywhere and probably dominated by the generative structure of the text itself. Semi-rare tokens seemed like a possible middle ground.

If some of those tokens correspond to more specific content, maybe plant names, ingredients, procedures, references or categories, then they could appear selectively between related folios. That was the initial intuition.

I also have the full token-by-token and page-by-page detail behind the graphs, so this is not only section-level statistics. I can trace exactly which rare tokens create each connection and between which folios. I think that part is probably more important than the global graph itself.