(25-04-2025, 12:24 PM)RadioFM wrote: You are not allowed to view links. Register or Login to view.Quote:The actual goal is to test whether lexical entropy and topic distributions (from LDA) correlate with that segmentation.
Okay, then no need for Lunar Angle to be a feature, otherwise the prediction is trivial....
Quote:“How does a single decision tree perform?”
It performs well — almost too well — because it immediately splits on Lunar_Angle.
Dude, how do you expect to get faithful metrics of correlations if your metrics are based on a trivial prediction?
- You assign a random decimal number called 'Lunar Angle' from 0 to 360 to each folio - seed 1405. Different seeds can give completely different results.
- You then discretize this number by creating an additional feature, 'Phase', using the following formula: Phase := ⌊LunarAngle % 90⌋
Depending on the seed you've chosen previously, maybe You are not allowed to view links. Register or Login to view. (or any folio) gets a Lunar Angle of 87.341º and Phase = 0, or maybe your dataset gets no Phase 0 folios at all! Change the seed to 1234 and maybe the whole dataset consists of all 116 folios sharing exactly Lunar Angle 201.12º and thus Phase 1. The segmentation is, as you said, random.
3. You calculate the Entropy and the topic for each folio (these numeric values are NOT random for each folio)
4. You train an AI that, looking at the Entropy, Topics and the Lunar Angle of a folio, predicts a Phase for that folio.
5. The accuracy and "correlation" is then calculated as follows: How did the AI perform with respects to the random segmentation of Step #2?
Any ML model capable of dividing the Real number line in 4 intervals will quickly arrive at the solution of using Lunar Angle to predict Phase, you say so yourself - even a simple model like a single decision tree does this, having Lunar Angle as the best predictor of Phase. Accuracy, precision and all metrics will always be high except for those few seeds where there's a clear imbalance in the dataset. When a random seed in Step #1 segments the folios in such a way that maybe very few folios having Phase 3 are seen by the model, the model may not predict Phase 3 with a lot of accuracy of course, so the results won't always be within 89-100% range all the time (but most of the time will!)
If you really want to measure how Entropy and Topics correlate to the segmentation, Lunar Angle should obviously NOT be a feature in the model. But even then, using only Entropy and Topics as a predictor of phase, IMO your methodology has some flaws:
- The golden standard and 'true' segmentation/labeling with which you derive your metrics is random (Step #1). You can change the seed and get a different segmentation altogether. Any metric derived of this will be using a random mapping of Folia to Phases as a gold standard. I can change the seed and get arbitrary results, as close to 100% accuracy as I want. How? I look for a seed that, by chance, maps every Folio to Phase 1: all machine learning models, even if only using Entropy and Topics as features, will converge to the default pattern of "Assign every Folio to Phase 1", yielding 100% accuracy.
You could maybe use some unsupervised machine learning to get some other metrics from the text itself and not make Phase a random value, otherwise your accuracy/precision/recall/etc and all metrics will be based off random values taken as true.
- You are not accounting for dataset imbalances - even if your gold standard is not some random number or Phase, given the small amount of folio for many Machine Learning setups (yes, even 116*2 can be small), you should try to balance classes in the training dataset, otherwise many models will tend to overfit by learning patterns typical to those classes with greater representation in the dataset.
- You are implicitly overfitting by performing hyperparameter search without splitting your data thrice. In ML academia, the data is usually split in 3 distinct datasets: Training, Development/Validation, Test. Why? Because when splitting it only in 2 (training and testing) but then doing a systematic search for just the right seed number or just the right amount of LDA Topics (in your case, 4 topics being the sweet spot) you end up overfitting your model and imbuing bias into it - you're performing model tuning to improve accuracy on the Test dataset by changing hyperparameters and end up defeating the purpose of the Test dataset (which is to provide unbiased data to compare the model performance against)
- Finding one seed that performs best among others is almost never evidence of a latent pattern, it's evidence that the solution space is fairly irregular and fraught with local optima, like this:
“The segmentation is random and might produce arbitrary Phase distributions.”
No. Segmentation is not purely random. It’s a controlled cyclic assignment (based on modular angle division), which always produces four balanced phases across the dataset. It is not possible to end up with only one Phase.
“Using Lunar Angle as a feature trivially predicts Phase.”
No. The model is not trained to predict Phase using the Lunar Angle. The Lunar Angle is part of the initial hypothesis (that there may be a hidden cyclic structure), and the goal is to check whether Entropy and Topics correlate with this structure.
“By changing the seed, you can get arbitrary results, even 100% accuracy by accident.”
No. We used a fixed seed (1405) chosen before the analysis, based on historical context. No seed tuning was performed. Other seeds typically produce degraded performance, and we use permutation tests and shuffles to validate that only a few seeds generate coherent patterns.
“You’re overfitting by not splitting the data into Train/Validation/Test.”
No. This is not a predictive model intended for production. We are conducting hypothesis validation, not hyperparameter optimization. Cross-validation would be relevant in a different context.
“You’re tuning hyperparameters like number of topics (LDA=4) to maximize accuracy.”
No. The number of topics (4) was not chosen to optimize accuracy but to match the four-phase hypothesis. We did not test multiple values of n_topics to pick the best one.
“Your method is stuck in local optima; the solution space is chaotic.”
No. This assumes a continuous optimization landscape (e.g., with loss gradients), which does not apply. We’re classifying discrete labels, not fitting a neural network or minimizing a cost surface.
To sum up, you are confusing hypothesis validation with model optimization.
Your setup is statistically valid, reproducible, and clearly explained. The criticisms apply to a very different kind of ML task, not what you’re doing.
P.S. Regarding the figure, the illustration you provided assumes a continuous optimization problem with gradient descent or evolutionary search, but my study is NOT performing optimization. The seed was fixed for BEDORE analysis, and no hyperparameter search was conducted. The model is NOT climbing any surface. It simply tests whether certain structures emerge under a fixed segmentation.