• Options
• Login
• Desktop Style

Automated Topic Analysis of the Voynich Manuscript

• Index

  • The Voynich Ninja
  • Voynich Research
  • Analysis of the text
  • Automated Topic Analysis of the Voynich Manuscript
• Automated Topic Analysis of the Voynich Manuscript

• RE: Automated Topic Analysis of the Voynich Manuscript

  ReneZ > Today, 12:52 AM

  Very interesting plots!
  
  When you colour-code a dot based on A vs B language, on what exactly do you base the choice A vs B?
  Is it according to Currier's list?

• RE: Automated Topic Analysis of the Voynich Manuscript

  MarcoP > 7 hours ago

  I would be curious to see PCA versions of the two plots in #83. I wonder if individual samples can be labeled with something like folio.parNum. Making individual sections visible (Herbal a/b, Pharma, Q13, Q20 etc) could help getting a feeling of what is happening
  
  Also, this clearly overlaps with the research by Lisa Fagin Davis and Colin Layfield that will soon be published
  
  EDIT: You are not allowed to view links. Register or Login to view.
  
  

  u/Miseryy Wrote:Your UMAP is massively overfit. In general tight strings that curve around are just indicative of a very small # neighbors used and too small distance threshold. You can replicate this effect with ~any dataset.
  
  Also, I'm of the opinion you should never do clustering on UMAP, ever. Furthermore "UMAP clustering" isn't a noun that exists. UMAP can be used as an initial preprocessing step, and then a standard clustering algorithm can be used. But again, I think it's terrible methodology, since you can ~always tune UMAP to achieve the clusters you want in the first place. People do it though, no denying that.
  
  I'd suggest going with the default parameters unless you really know what you're doing and have a good justification (read: mathematical reason) to adjust them. The parameters affect the math.
  
  Don't focus to much on meaning. If you want meaning, use PCA, and look at the vectors. The only interpretable meaning of UMAP is relative positioning. And even that is sketchy. You really should be taking away: there are groups that can be visually separated and appear to be distinct. UMAP is not proof of anything
  
  I would recommend that you just do clustering on your data. Not "UMAP clustering". How about starting with a simple hierarchical clustering and then looking at what you get? You can cluster either across genes or samples, and observe what falls into what group.

• RE: Automated Topic Analysis of the Voynich Manuscript

  quimqu > 1 hour ago

  OK, so after all the morning investigating, finally I got it clear.
  
  First: headlines topic distribution finally scored better NMF model, which gives discrete values for each topic. LDA topic distribution (the one that scored better in the Voynich) gives a percent on each topic, that adds up to 1. That's why both plots were so different (clouds for headlines NMF, sneaky fiures for Voynich LDA)
  
  Remember that the plots indicate the topic distribution per paragraph, and that the topic distribution is given by the model. So for K=2, UMAP is exploding the differences of a single line (from distributions (0,1), (0.01,0.99)... (1,0), accentuating the differences of topic distribution.
  
  To understand it better, I recalculated everything to K=3 topics with LDA (the second best score). For K=3 the vectors would look like (0,0.99,0.01) and UMAP reduces them to a bidimensional vector. I keep UMAP because it helps visualize the distribution of the topics assigned in my opinion more clearly as PCA. For this post I changed a bit the parameters so they are better separated. I post also PCA plots, so you can see the differences ploting UMAP vs. PCA.
  
  UMAP for K=2 (model finds 2 topics) and by paragraph:
  
     
  
  PCA for K=2 by paragraph:
  
     
  
  UMAP for K=3 (model finds 3 topics) and by paragraph:
  
     
  
  PCA for K=3 by paragraph:
  
     
• Next Oldest Next Newest