Marke Fincher's Word Pair Permutation Analysis

[MichelleL11]

I don't recall what was different about Fincher's paper - if there is no response from others tomorrow I will read it again.

Well, the major difference was that rather than looking for strings of matching words, Fincher looks at pairs of words that are separated by stepwise increases in word number (D).  In other words, the words in-between the "pairs" do not need to match.

Then he looks at (1) number of unique pairs found as plotted against D (both absolute and as a proportion of max amount seen); (2) the number of unique pairs that exist in both initially paired order AND in reverse order (again, both absolute and as a proportion of max amount seen); (3) the average squared frequency plotted against D and (4) the average squared difference plotted against D.

He gives "and the" as an example of a pair that is quite high at D=1 in English but is quite low in reverse order (little to no "the and").

Each of the plots provides a particular pattern that is relatively repeated within each of the natural languages (he provides 7 and does state he has tested many other languages with similar results . . . I'm a little uncomfortable with that statement, given recent events).

But, in any case -- the VM does look significantly different.  Bonus points for him for only testing language B with no labels.

In particular, he provides only the "proportion of max amount" graphs for both (1) and (2) and the VM does look much different in both cases.  For (1, forward pair),  there is a much lower proportion of maximum at D1 than any of the other languages for forward pairs and it peaks out (starts looking "randomly associated") much, much quicker than any of the other languages e.g., at basically D2!  The natural languages have slow ramps toward flat lines that start to be really flat around D9-D10 or so -- the VM is missing that ramp completely.

For (2, reverse pair) the VM looks even more weird, with again, a much lower proportion of maximum at D1 and then bizarre (but admittedly small) ups and downs with a weird peak of no relationship at word pair distance 12 (LAAU effects?).  Again, the ramp up to "random" shown by natural languages between D1-D6 or D7 for reverse pairs is completely missing.

Not unsurprisingly, the Average Squared Differences graph also shows how different the VM looks (as this plotting just amplifies the differences) with the VM being the lowest values shown (reminded me a bit of second degree entropy -- but not quite so drastically lower).  Fincher states this measurement best shows "order preference".  It is interesting that Latin and the VM are the closest in the ASD value that it "settles on" but it remains that the really huge swoop down from the early D1 to D9, 10 that Latin (and all the other natural languages) show is of course missing again for the VM.

He concludes that the VM differs in scale, scope, and character from natural languages using his measurements, and I find myself agreeing (although trying to think critically -- what is missing?).  This is the basis for the conclusion that some sort of transposition must occur to get it to "natural language."  He does go on to show that anachronistic polyaphabetic substitution ciphers like Vigenere  would only impact scale but not the character of the results -- so are not the cause of the differences. 

But, interesting -- the results, in particularly the ASD measurement, is NOT a perfectly flat line, which would be expected if whatever transposition process had completely randomized the word order -- so it seems there is some order still there.  In this way this again reminds me of the second order entropy issue -- what transpositions would cause the VM to look like this?

I appreciate you looking at the paper again and am looking forward to your or any board members thoughts.