29-06-2026, 11:29 PM
(29-06-2026, 08:34 PM)Mauro wrote: You are not allowed to view links. Register or Login to view.what you say is strictly true, but in practice the statistics of a text are (barring rare outliers, as the famous English novel written without a single 'e') mostly governed by the underlying language (which includes, of course, its specific orthography) and not by the text. Once a language has been chosen it's usually possible to say if a given text is written in that language or not just by checking the most basic statistics.
I insist that this claim holds only for texts of the same type, namely texts that consist mostly of discursive or narrative prose -- like novels, chronicles, philosophical treatises, newspaper and magazine articles, even textbooks. Like all the books in your example.
You cannot count on this claim if the text has a peculiar structure with little free prose, even if whatever is there is perfectly grammatical. Like a ship's log book, a terse list of saints, recipes, or kings, a catalog of products, a gazetteer, a book of exercises of algebra...
The most common words in such texts can be very different from those in discursive texts. Otherwise common words like "the", "of", "and" may occur scarcely or never. And any statistics of characters and bigrams are largely determined by their presence or absence in the few most common words in the text.
And, of course, character-based statistics are totally dependent on the orthography. You may say that Italian with any orthography other than the standard one "is not Italian". Even if that was a reasonable view in other contexts, it is not a useful position when trying to identify the language of a text with a completely original alphabet. Even if the VMS is a transcription of I Promessi Sposi, it will not be in the standard orthography...
All the best, --stolfi