24-08-2025, 07:24 AM
(This post was last modified: 24-08-2025, 08:19 AM by obelus.
Edit Reason: wordsmithery
)
These results are intriguing, but I remain pseudo perplexed about how the 11 topics "emerged," and whether their correlations with more-familiar categories require any explanation.
Comment:
The association that you find between "optimal topic" and scribal-hand categories excels over the Currier categories not because of its lower p value (both nulls are rejected at any sane cutoff), but because of the stronger clustering of counts in the contingency table, which is sufficiently convincing by inspection: in You are not allowed to view links. Register or Login to view., the table for scribal hand is clearly more contrasty than the one for Currier language. For You are not allowed to view links. Register or Login to view., a quantitative measure of effect size might help; conventional for these data would be "Cramér's V."
Question:
How was the text sample partitioned for classification? The number of text blocks tagged as paragraphs in the RF transliteration is less than 300. Summing counts in your contingency tables, 891 "paragraphs" were sampled in You are not allowed to view links. Register or Login to view., and 506 + 396 = 902 in You are not allowed to view links. Register or Login to view.. Am I right to conclude that Automated Topic Analysis is able to distinguish between 11 different topics using a sample size of 34 000 words / 900 paragraphs = ~38 words/paragraph?
(Caution: The p value is an intrinsically confounded measure because it always decreases with increasing sample size, while effect size is independent of sample size. Thus it is notoriously easy to mine "rock solid" correlations simply by gathering large data sets. While formally valid, such effects are often so small as to be meaningless in practice.)
Comment:
(21-08-2025, 08:19 PM)quimqu wrote: You are not allowed to view links. Register or Login to view.A lower p-value (approaching 0) indicates stronger statistical association between the topic distribution and the given variable.It is good practice to report the probability that a statistical inference is due to chance. According to your calculations, this risk is astronomically low for correlations between the 11 "optimal" topics and either Currier language or scribal hand. But the p value only quantifies confidence that a correlation exists, not the strength of correlation. The interesting part of a statistically significant result is its effect size, which is measured by other means.
The association that you find between "optimal topic" and scribal-hand categories excels over the Currier categories not because of its lower p value (both nulls are rejected at any sane cutoff), but because of the stronger clustering of counts in the contingency table, which is sufficiently convincing by inspection: in You are not allowed to view links. Register or Login to view., the table for scribal hand is clearly more contrasty than the one for Currier language. For You are not allowed to view links. Register or Login to view., a quantitative measure of effect size might help; conventional for these data would be "Cramér's V."
Question:
How was the text sample partitioned for classification? The number of text blocks tagged as paragraphs in the RF transliteration is less than 300. Summing counts in your contingency tables, 891 "paragraphs" were sampled in You are not allowed to view links. Register or Login to view., and 506 + 396 = 902 in You are not allowed to view links. Register or Login to view.. Am I right to conclude that Automated Topic Analysis is able to distinguish between 11 different topics using a sample size of 34 000 words / 900 paragraphs = ~38 words/paragraph?
(Caution: The p value is an intrinsically confounded measure because it always decreases with increasing sample size, while effect size is independent of sample size. Thus it is notoriously easy to mine "rock solid" correlations simply by gathering large data sets. While formally valid, such effects are often so small as to be meaningless in practice.)