21-09-2026, 06:25 AM
@ Stolfi
No, it’s not as simple as you’re trying to make it out to be.
Your own table doesn’t refute my argument; it continues to show exactly the connection: Even outside your SHDY group, tokens containing “sh” are followed by “qo” in about 29% of cases, compared to about 15% for tokens without “sh.” That’s a factor of 2. At the same time, it shows that the rest of the token structure is relevant. The two aren’t contradictory.
My actual point is the difference between your two comparisons: Your Pinyin table examines the last letter of one syllable and the first of the next—I’m aware of that effect, and we’ve discussed it before. My finding, on the other hand, concerns a character further back in the token and the beginning of the following token. Your Pinyin table therefore does not answer this question either.
My analysis based on token frequency has already shown that the effect isn’t limited to just a few high-frequency forms. The fact that it also occurs in rare forms doesn’t directly rule out a linguistic explanation, but it does require an explanation that goes beyond individual high-frequency word pairs and which, as far as I’ve been able to verify (again, I don’t speak Chinese), would be atypical for Chinese.
Of course, such correlations can also arise from grammatical constructions. Only “Shedy” is among the high-frequency words in the VMS.
And by the way: If qo is supposed to mean “and,” you’ll also have to explain why, in over 80 percent of cases, the following glyphs begin with k or t. That doesn’t quite add up either.
And as I’ve already hinted at, this isn’t the only long-range effect that extends beyond the word boundary and a space.
Here, too, these differences persist even after removing the most frequent token pairs. So the repetition of a few identical phrases isn’t enough to explain this on its own.
However, the mere existence of statistical dependencies in Chinese does not explain these specific patterns. To do so, you’d have to demonstrate corresponding dependencies in a fixed transcription of a Chinese reference text—particularly between characters preceding a consistent ending and the beginning of the next token. Based on my research, it’s extremely unlikely that you’ll find anything like that. But then again: I don’t speak Chinese.
No, it’s not as simple as you’re trying to make it out to be.
Your own table doesn’t refute my argument; it continues to show exactly the connection: Even outside your SHDY group, tokens containing “sh” are followed by “qo” in about 29% of cases, compared to about 15% for tokens without “sh.” That’s a factor of 2. At the same time, it shows that the rest of the token structure is relevant. The two aren’t contradictory.
My actual point is the difference between your two comparisons: Your Pinyin table examines the last letter of one syllable and the first of the next—I’m aware of that effect, and we’ve discussed it before. My finding, on the other hand, concerns a character further back in the token and the beginning of the following token. Your Pinyin table therefore does not answer this question either.
My analysis based on token frequency has already shown that the effect isn’t limited to just a few high-frequency forms. The fact that it also occurs in rare forms doesn’t directly rule out a linguistic explanation, but it does require an explanation that goes beyond individual high-frequency word pairs and which, as far as I’ve been able to verify (again, I don’t speak Chinese), would be atypical for Chinese.
Of course, such correlations can also arise from grammatical constructions. Only “Shedy” is among the high-frequency words in the VMS.
And by the way: If qo is supposed to mean “and,” you’ll also have to explain why, in over 80 percent of cases, the following glyphs begin with k or t. That doesn’t quite add up either.
And as I’ve already hinted at, this isn’t the only long-range effect that extends beyond the word boundary and a space.
Here, too, these differences persist even after removing the most frequent token pairs. So the repetition of a few identical phrases isn’t enough to explain this on its own.
However, the mere existence of statistical dependencies in Chinese does not explain these specific patterns. To do so, you’d have to demonstrate corresponding dependencies in a fixed transcription of a Chinese reference text—particularly between characters preceding a consistent ending and the beginning of the next token. Based on my research, it’s extremely unlikely that you’ll find anything like that. But then again: I don’t speak Chinese.