29-06-2026, 08:10 PM
(This post was last modified: 29-06-2026, 08:14 PM by Jorge_Stolfi.)
(29-06-2026, 07:02 PM)Mauro wrote: You are not allowed to view links. Register or Login to view.Indeed it's almost always possible to determine the language of an unknown text just by comparing basic statistics: for example, this is the output of a program I wrote some time ago which categorizes texts according to their statistics. Here I'm comparing a book in Italian ("L'amore di Loredana") with a panel of texts in different languages, by calculating the root-mean-square distance of just the bigrams distributions.
Generally true, ... provided that the unknown text is written in the "official" orthography, and contains a sufficiently large fraction of "normal" prose text. In that case, even the letter frequencies could distinguish Italian from English.
But the VMS is definitely not written in the official orthography of any language. Even if it was in un-encrypted Italian, it would be in an orthography that is not simple letter-by-letter mapping of the "official" one. It would use its own alphabet and digraphs.
And it would probably have its own quirks of spelling, like attaching the articles to nouns, detaching oblique pronouns from the verbs, marking stressed vowels in a different way, making heavy use of abbreviations (like "cãtaaŕ" for "cantare") etc.
And the text may make heavy use of certain words that are rare in typical novels and contain unusual digraphs. Like, a Medieval Italian herbal may have an excess of occurrences of the digraph "rb", and an excess of word-initial "h", if it repeats the old word "herba" often enough.
All the best, --stolfi