Can LAAFU effects be modeled?

[Jorge_Stolfi]

For comparison, here are the analogous lists for a Chinese text. 

And here is the same analysis for another Mandarin Chinese text.  The source is the 18th-century novel Dream of the Red Mansion by Cáo Xuěqín. As before, for each N in {2,3,4}, the first 10'000 N-phrases (sequences of N consecutive words) were extracted, then the occurrences of each distinct N-phrase were counted, and the top 20 for each N were listed.  Here "word" means a single Chinese letter (= syllable).  Punctuation was ignored, but phrases spanning parag breaks were excluded. 

The Chinese letters in the original file were represented in the old GB2312 encoding.  They were transcoded to Unicode by a Linux utility (autogb).  The Pinyin and English equivalents were obtained through Google Translate.  They are very approximate since the N-phrases often contain only half of a two-character (two-syllable) compound word, whose meaning may be quite unrelated to that of its individual characters.

N = 2:

    52 雨.村       | Yǔcūn              | Yucun                         
    46 士.隐       | Shì.yǐn            | Shiyin                         
    24 笑.道       | Xiào.dào           | Laughed                       
    19 道.人       | Dàoren             | Taoist                         
    17 子.兴       | Zǐ.xìng            | Zixing                         
    16 不.知       | Bùzhī              | I don't know                   
    14 了.一       | Le.yī              | One                           
    14 如.今       | Rújīn              | As now
    13 封.肃       | Fēng.sù            | Fengsu                         
    13 有.一       | Yǒuyī              | There is                       
    13 那.僧       | Nà.sēng            | That monk                     
    12 不.能       | Bùnéng             | Can't                         
    11 原.来       | Yuánlái            | Originally                     
    10 一.个       | Yīgè               | One                           
    10 不.过       | Bùguò              | But                           
    10 两.个       | Liǎng.gè           | Two                           
    10 二.人       | Èr.rén             | Two                           
    10 听.了       | Tīn.le             | After hearing it                 
    10 心.中       | Xīnzhōng           | In his heart 
   
N = 3:   

     6 士.隐.听    | Shì.yǐn.tīng       | Shiyin listened               
     6 子.兴.道    | Zǐ.xìng.dào        | Zixing said                   
     6 那.道.人    | Nà.dàoren          | That Taoist                   
     5 女.学.生    | Nǚ.xuéshēng        | Female student                 
     5 段.故.事    | Duàn.gùshì         | A story                       
     5 笑.道.你    | Xiào.dào.nǐ        | Laughed                       
     5 道.人.道    | Dàoren.dào         | The Taoist said               
     5 隐.听.了    | Yǐn.tīngle         | Shiyin listened               
     4 世.人.都    | Shìrén.dōu         | Everyone knows                 
     4 个.儿.子    | Gè'er.zǐ           | A son                         
     4 了.士.隐    | Le.shì.yǐn         | Shiyin                         
     4 了.雨.村    | Le.yǔcūn           | Yucun                         
     4 人.都.晓    | Rén.dōu.xiǎo       | Everyone knows                 
     4 原.来.是    | Yuánlái.shì        | It turned out to be           
     4 家.娘.子    | Jiā.niángzǐ        | The lady of the family         
     4 忘.不.了    | Wàng.bùliǎo        | Unforgettable                 
     4 政.老.爷    | Zhèng.lǎoyé        | Master Zheng                   
     4 晓.神.仙    | Xiǎo.shénxiān      | Knowing the immortals         
     4 林.如.海    | Línrúhǎi           | Lin Ruhai                     
     4 甄.家.娘    | Zhēnjiāniáng       | Mother of the Zhen family 
     
N = 4:     

     5 士.隐.听.了  | Shì.yǐn.tīngle     | Shiyin listened               
     4 世.人.都.晓  | Shìrén.dōu.xiǎo    | Everyone knows                 
     4 人.都.晓.神  | Rén.dōu.xiǎo.shén  | Everyone knows about gods     
     4 晓.神.仙.好  | Xiǎo.shénxiān.hǎo  | They know that gods are good   
     4 甄.家.娘.子  | Zhēn.jiā.niángzǐ   | The Zhen family lady           
     4 空.空.道.人  | Kōngkōng.dào.ren   | The Taoist Kongkong           
     4 那.道.人.道  | Nà.dào.ren.dào     | The Taoist said               
     4 都.晓.神.仙  | Dōu.xiǎo.shénxiān  | Everyone knows about gods     
     4 雨.村.笑.道  | Yǔcūn.xiào.dào     | Yucun laughed                 
     3 一.僧.一.道  | Yī.sēng.yīdào      | A monk and a Taoist           
     3 一.段.故.事  | Yīduàn.gùshì       | A story                       
     3 两.个.儿.子  | Liǎng.gè.er.zǐ     | Two sons                       
     3 了.世.人.都  | Le.shìrén.dōu      | Everyone knows about gods     
     3 了.两.个.儿  | Le.liǎng.gè.er     | They have two sons             
     3 仙.好.只.有  | Xiān.hǎo.zhǐyǒu    | Only gods are good             
     3 子.兴.笑.道  | Zǐ.xìng.xiào.dào   | Zixing laughed                 
     3 朝.代.年.纪  | Cháodài.niánjì     | Dynasty and age               
     3 灵.秀.之.气  | Língxiù.zhī.qì     | A spiritual aura               
     3 神.仙.好.只  | Shénxiān.hǎo.zhǐ   | Only gods are good             
     3 那.僧.笑.道  | Nà.sēng.xiào.dào   | The monk laughed

Comments:

(1) The big difference between these two samples (VoA transcripts and novel) show that these statistics, like many others, are a property of the text more than of the language.

(2) Nevertheless, the nature of the repeating phrases is strikingly different.  For N=2, the English list is dominated by pairs of "function" words in the general sense (136 of.the, 73 in.the, 43 it.is, 30 and.the, 26 for.the, ...), and even in Latin most entries have at least one "function" word.  But Mandarin has few "words" that are strictly "function words".  Most "words" (characters) that could be translated into English function words are also elements of dozens of compounds with different grammatical functions. Mandarin does have articles, often dispenses with the verb "to be", etc.  Thus, in retrospect it is not surprising that the most common repeating phrases consist of "contents" words.

All the best, --jorge