The 'Chinese' Theory: For and Against

[Jorge_Stolfi]

[quote="JoJo_Jost" pid='91398' dateline='1789888176'But “sh” simply behaves differently—across all tokens ... “sh” is drawn to “qo.” [/quote]

NO, not "all tokens"!

Let

• SHDY be the set of all tokens that match the RE pattern r"^\[q]?[o]?[dkl]?she?(dy|ey)$"  -- that is, "shdy", "shedy", sheedy", "shey", "sheey", possibly preceded by "d", "k", "l", "qo","qok",etc.
  
• XSHY be the set of all other tokens that contain "sh", besides those in SHDY.
  
• NOSH be the set of all tokens that do not contain any "sh".
  
• QOWW be the set of all tokens that start with "qo".
  
• NOQO be the set of all tokens that DONT start with "qo".
  

Here are the counts of consecutive token pairs in my transcription of the SPS, combined into the above classes:

          QOWW   NOQO
  SHDY      97    110
  XSHZ     151    369
  NOSH    1062   6075
    
Thus, reading by rows:

• A SHDY token is followed by "qo" ~47% of the time.
  
• Any other token with a "sh" is followed by qo ~29% of the time.
  
• A token without "sh" is followed by a "qo" ~15% of the time.
  

Reading by columns:

• A "qo" is preceded by a SDHY token 7% of the time, by another "sh" token 12% of the time, and by a non-"sh" token 81% of the time.
  
• A non-"qo" token is preceded by a SHDY token 1.6% of the time, by another "sh" token 5.6% of the time, and by a non-"sh" token 93% of the time.
  

Thus, while "sh" tokens do seem to be attracted to "qo" tokens, some "sh" tokens are clearly more attracted than others.  

So you cannot say that the attraction to "qo" is a property of the "sh" alone.  It depends on the whole token.  

So your claim "sh is attracted to qo" is like saying that in English, tokens that contain "th" are attracted to tokens that begin with "i".  Even if that is true, it would be a pretty useless statistic.  Much more useful would be to say that in English, the tokens "this" and "that" and "there" and "it" and "he" and "she" are often followed by "is" or "isn't".

My partition of "sh" tokens above into SHDY and XSHY was just a crude guess.  If you want to understand what is going on, you should look closely at which tokens are most common before which "qo" tokens.  I bet that, while many of the former may happen to be "sh" tokens, it will not be a categorical thing -- the "qo"-lovers will probably include other non-"sh" tokens, and I bet that some "sh"-tokens will definitely avoid "qo"-tokens.

All the best, --stolfi