16-02-2026, 01:15 AM
(16-02-2026, 01:07 AM)Fontanellean wrote: You are not allowed to view links. Register or Login to view.(16-02-2026, 12:29 AM)Dunsel wrote: You are not allowed to view links. Register or Login to view.I understand what you're saying about bigram statistics being driven by common words and subject matter. That absolutely happens in natural languages. Your examples make sense in that context. But what I'm seeing here doesn’t look like normal topic drift. I'm not just saying that “ed is frequent.” I'm saying that a very high-frequency bigram simply disappears on many pages, and then appears heavily on others, within and across topics. That kind of page-level on/off behavior is what caught my attention.
If there’s a natural-language example where a very high-frequency digraph shows this kind of behavior, I’d genuinely be interested to see it. It would help clarify whether what we’re seeing here is unusual or not.
You don't even have to look at bigrams. Take an instruction manual with one page written in British English and another in Pinyin for Mandarin. "Z" will hardly appear at all on the former but will be all over the place in the latter.
Well, I suppose that's one explanation.


