The 'Chinese' Theory: For and Against

[Jorge_Stolfi]

My point was not about raw character frequencies! It was about a conditional dependency across a visible token boundary: the internal form before final y changes the probability that the next token begins with qo.

Yes! Because such correlations are not features of characters, but of words.  It is certain words that tend to be followed by certain other words; and the first happen to often end in y when the second happen to begin with qo.  

If qo is “and”, the distribution should mainly follow syntax or enumeration structure, not the internal shape of the previous token!!!

The following counts are from the herbal section proper of Culpeper's Herbal.  The first columns is the count of tokens that end with each letter.  The second column is the count of tokens that end with each letter and are followed by the word 'and'.

    22446 e   2030 s!and
    18097 s    736 e!and
    14166 d    633 d!and
    10477 t    574 r!and
     9366 n    501 t!and
     8464 r    436 y!and
     8128 h    389 n!and
     6821 y    378 h!and
     5406 f    303 g!and
     4493 o    200 l!and
     3807 g    180 k!and
     3411 l    136 m!and
     2127 a     54 b!and
     1770 m     49 o!and
     1448 k     48 p!and
      858 p     45 f!and
      773 w     38 w!and
      571 b     24 a!and
      284 i     23 x!and
      231 u      3 u!and
      223 x           
      110 °           
        9 v           
        3 z           
        2 '           
        2 c

As you can see, the word before an 'and' is almost three times more likely to end with "s" than with "e"; whereas, in general, a word is slightly more likely to end with "e" than with "s". 

Is that because of some mysterious rule that makes the word 'and' attract the letter 's'?  Not quite.  Here are the counts for the words that occur before the word 'and':

   262 vertues!and
    95 bruised!and
    87 wine!and
    83 liver!and
    79 them!and
    78 long!and
    62 july!and
    56 leavs!and
    52 stomach!and
    51 june!and
    50 courses!and
         ...
     1 yonger!and
     1 zedoary!and

(1645 different word types in all).  

While the count of 2030 for "-s" before the word "and" is due to 494 different word types, note that 262 of those occurrences (13%) are due to a single word type, "vertues", that is often followed by "and" -- because, in the entry of every plant, there is a sub-entry titled "Vertues and use" or "Vertues and uses".

On the other hand, the most common word types overall in that text are

 10158 the
  6780 and
  4362 of
  3045 in
  2274 or
  2242 to
  2213 it
  2009 is
  1924 a
  1267 with
   ...

Yet there is not even one occurrence of "the and" or "and and", and only a handful of "of and", "in and".

Thus it is not that the word "and" attracts the ending "-s"; rather, the word "and" attracts certain word types, like "vertues", and it just happens that many of these words end in "-s".

If y is a pitch marker, the model still has to explain why the material before y changes the following qo probability so strongly.  what is the concrete mechanism that connects the internal pre-y form with the next token’s qo?

Same answer.  

Saying "the internal pre-y form is connected with the next token’s qo" is an obfuscated way of saying "the frequency distribution of words before words that begin with qo is different from the frequency distribution of words in the text as a whole".  

In particular, the former distribution has more words that end in -y than the latter.  Why?  For the same reason that, in Culpeper's book, words that end in "-s" are more common before "and" than in the text as a whole.  Basically, "just because". 

All the best, --stolfi