14-11-2025, 12:33 PM
(This post was last modified: 14-11-2025, 12:34 PM by Jorge_Stolfi.)
And also:
6. Exclude any "weirdo" that occurs only a few times in the section in question. Exclude words that contain it, and lines that start with it.
7. Try to express frequencies as fractions (or percentages), rather than raw counts. I think the former are easier to reason about.
8. Be aware that frequencies f = k/n obtained by counting (like "frequency of d at start of line) have a sampling error σ that depends on the number of "trials" n (in that case, count of lines in the sample), the number of "successes" k (in that case, count of lines that start with d), as well as on the a priori expectations about f (which could be derived, for example, from the freq of d in word-initial position). I owe you the formula for σ. But the point is that "discrepancies" in the frequencies are significant only if they are 3 times σ or more.
9. Be aware also that if you compute a large number of frequencies (like the frequencies of 20 glyphs), some of them will come out anomalous (above 3σ) just by chance. Some of these false anomalies may be detected by splitting the sample text in half and running the analysis on each half separately. Those false discrepancies are unlikely to occur on both both halves.
All the best, --stolfi
6. Exclude any "weirdo" that occurs only a few times in the section in question. Exclude words that contain it, and lines that start with it.
7. Try to express frequencies as fractions (or percentages), rather than raw counts. I think the former are easier to reason about.
8. Be aware that frequencies f = k/n obtained by counting (like "frequency of d at start of line) have a sampling error σ that depends on the number of "trials" n (in that case, count of lines in the sample), the number of "successes" k (in that case, count of lines that start with d), as well as on the a priori expectations about f (which could be derived, for example, from the freq of d in word-initial position). I owe you the formula for σ. But the point is that "discrepancies" in the frequencies are significant only if they are 3 times σ or more.
9. Be aware also that if you compute a large number of frequencies (like the frequencies of 20 glyphs), some of them will come out anomalous (above 3σ) just by chance. Some of these false anomalies may be detected by splitting the sample text in half and running the analysis on each half separately. Those false discrepancies are unlikely to occur on both both halves.
All the best, --stolfi
