Interchangeable ch/sh and k/t characters

[Jorge_Stolfi]

Well, a 'deviant' text if ever there was one  . It's also short, 1812 words, which by itself tends to give unreliable results (~10K words would be better).

Agreed.  I am still looking for something better...

I removed/replaced the non-italian characters, i.e. Gesù -> Gesù.

That is is what happens if you open the file in the browser and an copy-paste it from there.  Our server for some reason tells the browser that it is ISO-Latin-1, but in fact it is Unicode UTF-8.  That is why I advised to download the file instead.   That word is indeed "Gesù" in the file. 

Beware that there are saint names with non-Italian characters, namely "ñ" ("San Raimondo di Peñafort.") and "á" ("San Josemaria Escrivá").  Removing those would be cheating...

However it's also true that the most similar texts according to the bigrams distribution are all Italian texts:

If I read the first table correctly, it says that the distance d(L,I) from the List to the Italian texts is about the same as the distance d(C,I) from the latter to Caesar's book; but the distance d(L,C) is a bit bigger that both.  

   

I think this demonstrates that at least the bigrams statistic is mostly determined by the underlying language and not by the content of the text (barring even more extreme cases).  I will add: the vocabulary of the list is extremely aberrant, in practice only variations of the word 'Saint', names of months, numbers from primo (first) to trentuno (thirty-one) and personal names/toponyms.

But that is my point. This List is "aberrant" because it is not a discursive/narrative text, like all the others.  It has a very different nature, purpose, and structure, that causes its word frequency distribution to be extremely different from that of those texts; and that in turn causes the bigram frequencies to be very different too.  And yet the language is definitely Italian, and even mostof the words (even most of the names) are specifically Italian.

The "moral of the story" is that, when one tries to identify the language of the VMS (or any other mysterious text) by any set of statistics, one should compare it to reference texts that hopefully have the same nature and vocabulary.  That of course is hard since we ignore even the general topic of most sections.  We can guess that narrative/discursive prose texts may be adequate references for the Bio section, but maybe not so much for the Herbal and Pharma; and probably very inadequate for Astro/Cosmo/Zodiac and the Starred Parags.

All the best, --stolfi