The 'Chinese' Theory: For and Against

[nablator]

How to build a babble detector? Not sure. Some compression algorithm run on all (short) substrings?

Okay, I wrote a quick babble detector with Byte Pair Encoding. The size of the string (with spaces) is divided by the size of the compressed string (without dictionary) to get the compression factor.

Looking for 4 consecutive words with BPE compression factor >= 4.

3 in chovynoise:
nikd jheuf nikd jheuf
hitd hitd gop gop
ret sef ret sef

23 in VMS Q20:
qokaiin shey qokaiin shedy (error in RF1b, should be qokaiin shey qokain shedy on line f103r.20)
chedy qokeey qokeey lchedy
okaiin shedy shedy qokaiin
shey l shey lshey
qodain shedy chedy qodain
okeeey qokeeey okeey okeey
otshedy qokchdy qotshdy qokchdy
kchedy shdy kchedy chedy
oteey oteey lkain okain
okedy qokchedy chedy qokedy
okeey qokeey qokeedy qokeey
otedy qokedy qokeedy qokeedy
lkedy lkeedy chedalkedy lkeedy
okeedy oteedy qokeedy okeedy
oteedy qokeedy okeedy okeedy
aiin cheey qokeey qokeeaiin
qokeedy qokeedy chey qokeedy
qokeedy qokeedy qokeedy qotey
qokeedy okeedy qokey qokeedy
shey qokear shey qokeey
ateey qokeedy okeey qokeedy
chedy qotain chody qotain
sheey qokey qokchey qokchey

Looking for 5 consecutive words with BPE compression factor >= 4.

2 in chovynoise (actually it's twice the same "nikd jheuf nikd jheuf" string):
ro nikd jheuf nikd jheuf
nikd jheuf nikd jheuf vepd

18 in VMS Q20: 
shckhy qokaiin shey qokaiin shedy
qokeey chedy qokeey qokeey lchedy
otchedy qodain shedy chedy qodain
qokaiin chey chokaiin chear ar
aiin otal taiin qokaiin otal
okair chtl lkaiin okair chtl
qokedy okar chdy okar char
qokeedy qokeedy lchdy oteedy lchedy
qokeedy lchdy oteedy lchedy qokeedy
lchdy oteedy lchedy qokeedy qokeedy
lchedy qokeedy qokeedy qokeo lchedy
okeedy oteedy qokeedy okeedy okeedy
qokeedy qokeedy chey qokeedy qokedy
qokeedy qokeedy qokeedy qotey qokeey
qokain okeiin chey qokain okeey
ateey qokeedy okeey qokeedy qoky
qo qokain sheckhy qokain shekain
qokain sheckhy qokain shekain shkain

Looking for 6 consecutive words with BPE compression factor >= 4.

0 in chovynoise

15 in VMS Q20:
shdy qokeey chedy qokeey qokeey lchedy
odaiin otchedy qodain shedy chedy qodain
chkeiin okair chtl lkaiin okair chtl
qokchedy chedy qokedy okar chdy okar
qokeey qokeedy qokeey chedal chedy qokeey
qokeedy qokeedy lchdy oteedy lchedy qokeedy
qokeedy lchdy oteedy lchedy qokeedy qokeedy
lchedy qokeedy qokeedy qokeo lchedy qokey
qokeedy qokeedy qokeedy qotey qokeey qokeey
qokeedy qokeedy qotey qokeey qokeey otedy
qochey qokeey chey teey qokeedy qokeedy
qoaiin chedy qotaiin chety laiin chedy
shedain qokeedy chodain otedain qokeedy qokeedy
qo qokain sheckhy qokain shekain shkain
qokain sheckhy qokain shekain shkain shedy

Note: factor 4 is a bit too much, it favors exact repetitions.

For example "qokaiin shey qokain shedy " is compressed to "WiSRWSdR", factor = 26/8 = 3.25
Z = "qo"
Y = "Zk"
X = "Ya"
W = "Xi"
V = "n "
U = "Vs"
T = "Uh"
S = "Te"
R = "y "

VMS Q20: 406 occurrences of 4 consecutive words with BPE compression factor >= 3.

chovynoise: 5 occurrences of 4 consecutive words with BPE compression factor >= 3.

VMS Q20: 403 occurrences of 10 consecutive words with BPE compression factor >= 3.

chovynoise: 2 occurrences of 10 consecutive words with BPE compression factor >= 3.