Interchangeable ch/sh and k/t characters

[Mauro]

You are not allowed to view links. Register or Login to view. is a list of the 366 Catholic patron saints of the day in Italian.  The file main.txt = main.src has one saint per line. The file main.wds has one word per line; use only lines that begin with 'a '.  The character '°' denotes abbreviation or truncation, as in "Sant°"  The character '~'  is hyphen.  Both files are in Unicode UTF-8 encoding.  You should download them; opening in the browser and copy-pasting the contents will not work.

This file is too terse; more succinct than what the VMS parags are likely to be.  It would be better if there was a bit more information about each saint, like "martire" and "secolo XV" and what is its "specialty" (like, here in Brazil Santo Antônio is patron of marriages and the saint to pray to for help finding a misplaced or lost item.)

And I really should tell the story of Santo Expedito, but don't have the time now.

All the best, --stolfi

Well, a 'deviant' text if ever there was one  . It's also short, 1812 words, which by itself tends to give unreliable results (~10K words would be better). I removed/replaced the non-italian characters, i.e. Gesù -> Gesù.

Anyway, the first conclusion is the list of Saints is surely statistically aberrant from other Italian texts and the distances do not reach the (empirical) threshold I use to determine if a text is written in a certain language. So this is good for you!

However it's also true that the most similar texts according to the bigrams distribution are all Italian texts:

   

The table is ordered by distance from the list of Saints. Texts from 2 to 15 are in Italian, 16 and 17 are Latin, 18 is (old) French, from 19 to 23 English, 24 is German (with diacritics removed) and 25 is Classic Greek (in greek characters, so an outlier).

The list of Saints is not particularly close to Italian (yellowish green colour, while all the other texts in the same language make nice green squares), but surely Italian is closest to the list of Saint than all other languages. So even with an extremely 'deviant' text Italian stands out as the closest relationship. I think this demonstrates that at least the bigrams statistic is mostly determined by the underlying language and not by the content of the text (barring even more extreme cases). 


I will add: the vocabulary of the list is extremely aberrant, in practice only variations of the word 'Saint', names of months, numbers from primo (first) to trentuno (thirty-one) and personal names/toponyms. And indeed it shows:

   

So by this statistic, indeed, the text is unrecognizable.

Note: the texts in this last table are the same as before (excep the Greek one) but they are numbered differently.