Interchangeable ch/sh and k/t characters
#20
(29-06-2026, 08:10 PM)Jorge_Stolfi wrote: You are not allowed to view links. Register or Login to view.
(29-06-2026, 07:02 PM)Mauro wrote: You are not allowed to view links. Register or Login to view.Indeed it's almost always possible to determine the language of an unknown text just by comparing basic statistics: for example, this is the output of a program I wrote some time ago which categorizes texts according to their statistics. Here I'm comparing a book in Italian ("L'amore di Loredana") with a panel of texts in different languages, by calculating the root-mean-square distance of just the bigrams distributions.

Generally true, ... provided that the unknown text is written in the "official" orthography, and contains a sufficiently large fraction of "normal" prose text.    In that case, even the letter frequencies could distinguish Italian from English.

But the VMS is definitely not written in the official orthography of any language. Even if it was in un-encrypted Italian, it would be in an orthography that is not simple letter-by-letter mapping of the "official" one.  It would use its own alphabet and digraphs. 

And it would probably have its own quirks of spelling, like attaching the articles to nouns, detaching oblique pronouns from the verbs, marking stressed vowels in a different way, making heavy use of abbreviations (like "cãtaaŕ" for "cantare") etc.

And the text may make heavy use of certain words that are rare in typical novels and contain unusual digraphs. Like, a Medieval Italian herbal may have an excess of occurrences of the digraph "rb", and an excess of word-initial "h", if it repeats the old word "herba" often enough.

All the best, --stolfi

Yes of course, but what you're saying now contradicts your previous position:
Quote:Anyway, let me insist again that statistics (glyph and word frequencies, Zipfness, entropies, correlations etc) are not properties of languages, but of texts.

If I can summarize: what you say is strictly true, but in practice the statistics of a text are (barring rare outliers, as the famous English novel written without a single 'e') mostly governed by the underlying language (which includes, of course, its specific orthography) and not by the text. Once a language has been chosen it's usually possible to say if a given text is written in that language or not just by checking the most basic statistics. Instead the text itself has little relevance, even when comparing very diverse texts as in my example of You are not allowed to view links. Register or Login to view.. A Medieval Italian herbal might have an excess of "rb" and some orthographical quirks, but I bet it'd be recognized as Italian nonetheless. If you have a transcription in .txt format or the like I'd be glad to check.
Reply


Messages In This Thread
RE: Interchangeable ch/sh and k/t characters - by Mauro - 29-06-2026, 08:34 PM

Forum Jump:


Users browsing this thread: 1 Guest(s)