The 'Chinese' Theory: For and Against

[Jorge_Stolfi]

But in the SBJ the variability of the 'zhǔzhì' word ( 主 治 ) is -0.42. ( Using your bencao-fu.pyj as my source, and which gives 'zhǔ zhì' as one word and not as two words. )

Let me address first the part of your message that is about the SBJ only.  I'll leave the parts about the SPS for another message.

The joining of Chinese syllables into two- or three-syllable compounds is a thing that exists only in pinyin, mean to help translation of the text into Western languages, which operates with the compounds rather than isolated syllables.  In the Chinese script the characters are written one after another without any spaces or grouping into compounds.  Native speakers will "know" when two or more characters have a specialized meaning that is not implied by them in isolation.  Just like an English speaker knows that "pen drive" is not how sharpened quills commute to work.

Thus you should use the file "in/bencao-fu.utf" (Chinese characters in Unicode UTF-8) as your SBJ source.  If you are doing the analysis in python3, you can read that file by saying "stdin.reconfigure(encoding='utf-8')" at the beginning (or for any input file instead of "stdin").  Then each Chinese character will be read as a single string character. 

Then, if your python3 source file is in UTF-8 too, you can ask "if ch == '主': ..." or "if line[i:i+2] == '主 治': ..." or "if re.search(r'主 治', line):...".  And you can remove Chinese punctuation with "line = re.sub(r'[　。，]', '', line)"  (Beware that the first character inside the brackets is Unicode's IDEOGRAPHIC SPACE, which this 屎 forum editor sometimes decides to delete.)
 
Now, if you look at that "rooster" recipe, it can be broken down into the following fields:

   

The translations should be taken with a metric ton of salt (Google translate will say "burns", "cold sores", "fever", etc. for the same character, depending on what is around it.)

As you can see, the SBJ is not consistently using 主 治 = "main use(s)" as the keyword to start a list of uses.  In two sub-recipes it says just 主 = "mainly [for?]".  Presumably because what follows the key is a "verb-like" character ("kills", "drains") rather than a "noun-like" one like in the other cases. Thus counting just 主 治 is not right.  And in the sub-recipe J the SBJ uses neither; perhaps because that entry starts with "can ..."

Advancing on the next message, the VMS column on lines C-I shows the words that I propose correspond to 主 治 and/or 主 in the recipe.  The "translation" of both seems to be daiin or a "misspelling" thereof.  The "-" means that the word appears as a suffix rather that a word by itself in my transcription file.  In the paper I did not propose any VMS key for line J, since the SBJ has no 主 there; but in fact there is an lkaiin apparently at the right spot. 

All the best, --stolfi