(29-06-2026, 06:14 PM)Jorge_Stolfi wrote: You are not allowed to view links. Register or Login to view.Anyway, let me insist again that statistics (glyph and word frequencies, Zipfness, entropies, correlations etc) are not properties of languages, but of texts.
You are strictly reason, of course, but it's also true that texts from the same language have, save rare exceptions, similar statistics.
Indeed it's almost always possible to determine the language of an unknown text just by comparing basic statistics: for example, this is the output of a program I wrote some time ago which categorizes texts according to their statistics. Here I'm comparing a book in Italian ("L'amore di Loredana") with a panel of texts in different languages, by calculating the root-mean-square distance of just the bigrams distributions:
As you can see, "Amore di Loredana", written in 1908 by a certain L. Zuccoli from Canton Ticino, clusters with the Italian texts (which have dates ranging from late middle ages to contemporary, with different genres including poetry, and include a specialized text: 'Appunti di anatomia', notes from anatomy lessons), and not with texts in other languages.
Note: DHR means 'declaration of human rights' (a very short text prone to statistical quirks)