17-11-2025, 03:47 PM
(17-11-2025, 12:29 PM)Rafal wrote: You are not allowed to view links. Register or Login to view.Now as we are comparing different texts in different languages we don't know the source of differences.
The good candidate would be Bible as we can easily get it in most languages.
Thanks Rafal, yes, this was one of the items on my list: search for the same text in different laguages (as much different, better).
(17-11-2025, 02:50 PM)Jorge_Stolfi wrote: You are not allowed to view links. Register or Login to view.The Chinese texts, in particular, use an old encoding with two bytes per
Chinese character. There is a Linux program "autogb -i gb -o utf8" that
converts them to the modern Unicode/UTF8.
Hello Jorge,
I have taken a look at the files. I think I can use them directly. I understand "a" is character and "p" is punctuaton, so I just need to put all "a" and "p" in a row to get the words and punctuation.
I will try to get also the Pentateuch in different european languages, so we can compare all of them in a single run.
Thank you both.

