• Options
• Login
• Desktop Style

Mapping Voynich connections through rare tokens

• Index

  • The Voynich Ninja
  • Voynich Research
  • Analysis of the text
  • Mapping Voynich connections through rare tokens
• Mapping Voynich connections through rare tokens

• RE: Mapping Voynich connections through rare tokens

  quimqu > 3 hours ago

  (5 hours ago)nablator Wrote: You are not allowed to view links. Register or Login to view.

  (Yesterday, 12:55 PM)quimqu Wrote: You are not allowed to view links. Register or Login to view.So I started wondering if the opposite approach might actually be more informative.

  
  Or you could use a balanced approach where rare terms are upweighted to reflect their relative importance.
  
  

  Quote:In addition to creating the A matrix as described, which uses straight TF values, two weighting schemes are also employed to modify the values contained in A. The two schemes applied are Term Frequency-Inverse Document Frequency (TF-IDF) and Log-Entropy (LE).

  You are not allowed to view links. Register or Login to view.

  
  I don’t think that Lisa's work is really the same thing as what I’m doing (it is simmilar to my automitic topic analysis). What they describe there is basically a weighting scheme inside an LSA/topic-modelling framework. TF-IDF increases the influence of rarer words and reduces the effect of very common ones. That is standard NLP.
  
  But their goal is still global semantic structure and document similarity through latent-space methods. They are trying to see whether nearby text behaves coherently, whether sections cluster, whether discourse segmentation exists, etc.
  
  My approach is much more explicit and local. I am not embedding the text into a latent semantic space or reducing dimensions. I am literally tracing the actual low-frequency tokens themselves across folios and sections, then building direct overlap networks from those tokens. In practice this changes the interpretation a lot.
  
  A TF-IDF weighted LSA model may tell you that two pages are “similar” in some abstract statistical sense (so do other topic models). What I’m looking at is closer to: “this exact semi-rare token appears in these particular folios and creates this concrete bridge between these sections”. So when You are not allowed to view links. Register or Login to view. or some Marginal Stars folios emerge as hubs, I can inspect the exact tokens creating those links page by page. That is a different scale of analysis.
  
  Also, interestingly, their own paper actually supports part of what I’m seeing. They found that Biological, Pharmaceutical and Stars sections show unusually high internal cohesion compared to other sections. That is not very far from the kind of structure emerging in my rare-token graphs.
  
  The important difference is that I’m not measuring coherence of adjacent text windows. I’m measuring selective reuse of uncommon vocabulary across distant folios.

• RE: Mapping Voynich connections through rare tokens

  quimqu > 3 hours ago

  (4 hours ago)Dunsel Wrote: You are not allowed to view links. Register or Login to view.If I strip the gallows from those hapax tokens, that decline flattens out.

  
  That is interesting, although I would probably be careful interpreting the gallows result too deeply for now.
  
  If removing gallows causes many forms to collapse into already existing tokens, then part of the flattening may simply come from reducing the vocabulary space artificially. The Voynich token space already has extremely small Levenshtein distances between many words, so removing one high-information element like gallows can easily create merges with forms that already exist elsewhere in the manuscript.
  
  My own objective was a bit different anyway. I was trying to see whether low-frequency tokens create bridges between folios and sections. That is why I did not use hapax legomena initially: a single occurrence cannot really create network structure or repeated connections between pages.

• RE: Mapping Voynich connections through rare tokens

  Dunsel > 3 hours ago

  (3 hours ago)quimqu Wrote: You are not allowed to view links. Register or Login to view.My own objective was a bit different anyway. I was trying to see whether low-frequency tokens create bridges between folios and sections. That is why I did not use hapax legomena initially: a single occurrence cannot really create network structure or repeated connections between pages.

  
  But, if one scribe creates a hapax in their work and then that word gets used or copied by another scribe, that creates a network between scribes.  And keep in mind there's more than just a few folios in herbal that are not currier A (Scribe 1). If you're testing all of herbal then you're mixing two regimes and will get a false network connecting to other scribe 2 pages in other sections.  Same for Scribe 1 and 3 in Pharma.

• RE: Mapping Voynich connections through rare tokens

  quimqu > 3 hours ago

  (3 hours ago)Dunsel Wrote: You are not allowed to view links. Register or Login to view.

  (3 hours ago)quimqu Wrote: You are not allowed to view links. Register or Login to view.My own objective was a bit different anyway. I was trying to see whether low-frequency tokens create bridges between folios and sections. That is why I did not use hapax legomena initially: a single occurrence cannot really create network structure or repeated connections between pages.

  
  But, if one scribe creates a hapax in their work and then that word gets used or copied by another scribe, that creates a network between scribes.  And keep in mind there's more than just a few folios in herbal that are not currier A (Scribe 1). If you're testing all of herbal then you're mixing two regimes and will get a false network connecting to other scribe 2 pages in other sections.  Same for Scribe 1 and 3 in Pharma.

  
  I did a quick graph of how the rare tokens are distributed by writting hand (according to EVA transliteration). There are plenty of connections, meaning that the use of the rare tokens is transversal between scribas.
  
     

• RE: Mapping Voynich connections through rare tokens

  Dunsel > 2 hours ago

  (3 hours ago)quimqu Wrote: You are not allowed to view links. Register or Login to view.I did a quick graph of how the rare tokens are distributed by writting hand (according to EVA transliteration). There are plenty of connections, meaning that the use of the rare tokens is transversal between scribas.

  
  And that's showing pretty much the same thing I was seeing with the hapax.  Scribe 1's connection weakens sequentialy from Scribe 2 to Scribe 5.  You're showing the 1-3 connection slightly stronger than the tests I run which is not unexpected and 2 & 3 are showing a strong connection.
• Next Oldest Next Newest