(21-09-2025, 08:44 PM)Jorge_Stolfi wrote: You are not allowed to view links. Register or Login to view.Just to clarify: in your analysis you take the parags (only, excluding labels and titles) of each page, do some cleanup, compute the token frequency distribution in those parags for each page, and then compare the distributions of different pages to identify "topics", by various criteria. Is that so?
Well, not exactly. Let me try to explain.
A topic model tries to compress a corpus into a small set of recurring word patterns and to say, for each text unit, how much of each pattern is present. In my case the text unit is a “paragraph”: in my case, pargraphs are the MS paragraphs, the lone lines outside the paragraphs and I also have merged labels and very short lines into their nearest text. My goal was to have a short group of words to compare topics. Then I turn each paragraph into a bag-of-words after cleaning (I remove single-grapheme tokens and a short empirical list of near-ubiquitous, non-discriminative forms; commas in the EVA runs I showed were treated as spaces, but that’s a switch I can flip; I will also take a look at your script and see how I can handle this; i am very aware of the commas issue that you explained several times).
The model learns two things: a set of word distributions (the topics themselves) and, for each paragraph, a vector of proportions telling how those topics are mixed. You choose K, the number of topics; too small and themes are merged, too large and they fragment.
Plain LDA or NMF will happily use topics to capture whatever explains variance, including orthography or scribal habits. That's why I saw that topics aligned to hands and language but seemed also to aligne with sections. But I needed a way to "clean" the scribal habits from them. DMR changes the default in a simple way that matters here: the topic vocabularies are shared by everyone, but the prior over a paragraph’s topic proportions depends on its metadata (language A/B, writing hand, Currier hand) through a small regression. In practice, style doesn’t redefine the topics; it mainly nudges how much of each topic a paragraph tends to use. That’s why, after cleaning, DMR yields topics that line up with sections, while language and hands show up as differences in prevalence rather than different word lists.
Each paragraph is modeled as a mixture of K topics. In standard LDA, the prior over those mixtures is the same for every paragraph. In DMR, that prior depends on the paragraph’s metadata (language A/B, writing hand, Currier hand), while the topic word distributions themselves stay shared across the corpus. I chose K by checking which value made the inferred topics align best with the manuscript sections (using AMI and the heatmaps/timelines); on this data, K=6 worked best.