Yesterday, 07:00 PM
(23-10-2026, 05:58 PM)npcompl33t wrote: You are not allowed to view links. Register or Login to view....Let me make two meta-comments about the methodology:
It is naive to expect that feeding a bunch of data points to a sufficiently sophisticated data analysis tool will provide real insight into the process that generated that data. The tool can only identify clusters, correlations alignments,etc. To translate those results into real understanding one must look hard at the process and the data, and try to infer how exactly one may have produced the other. In our case, finding that the digraph frequencies of two sections are different tells us nothing; we must look at the words that caused those digraph differences, and try to figure out what could have caused those word differences.
It is usually a bad idea to try to address too many questions at once. One often gets distracted by multiple hypotheses at the same time. If an open question has multiple parts, pick one part and try to solve it. Then go back and see whether the same solution works for the other parts. In our case, the statistical divergence between Herbal-A and Herbal-B probably has a different nature and different cause than the divergence between Herbal-A and Pharma. And the latter may well have different nature and causes than the difference between Bio and Stars. Thus, it is probably more effective to understand Herbal-A vs Herbal-B first, ignoring the other sections. One advantage there is that the general topic (herbal) seems to be the same, so the cause of the divergence must be something else.
Using more data variables than data points is asking for trouble. It is like fitting a 10th degree polynomial to seven XY points. Even with MDS/PCA, the results may be garbage. Since you have only 10 or so sub-sections, you should spend some time figuring out a meaningful way to reduce the raw digraph frequency data to only (say) 5--20 measurements. Like, by combining data pair counts into macro-counts such as "gallows to bench transitions", "tokens ending in n/in/iin/ir/iir/il", etc .
All the best, --stolfi

