21-09-2025, 09:45 PM
(21-09-2025, 09:28 PM)quimqu wrote: You are not allowed to view links. Register or Login to view.Then I turn each paragraph into a bag-of-words
Ah, ok - so your unit is the parag, not the page.
(21-09-2025, 09:28 PM)quimqu wrote: You are not allowed to view links. Register or Login to view.The model learns two things: a set of word distributions (the topics themselves) and, for each paragraph, a vector of proportions telling how those topics are mixed. You choose K, the number of topics; too small and themes are merged, too large and they fragment.
So you are trying several K-clustering algorithms on the same set of data points, where each point is a bag-of-words -- essentially an N-vector where N is the number of distinct word types. Where the algorithms do not assign each parag to a single cluster (topic), but gives it a "belonging" or "mixing" score for each cluster. Is that correct?
Can we interpret those scores as Bayesian probabilities of the parag belonging to each topic?
All the best, --jorge

