Experiments with language corpora

[DonaldFisk]

Very intersting Marco!

Have you done any pre-processing on the text files to 'clean them up' (as far as that is needed)?

Thank you, Rene!
No, I haven't done any pre-processing. I didn't convert upper-case / lower-case, nor remove punctuation. A difficulty is that several of the files are not written in the Latin alphabet, so a generic pre-processing system is not easy do define. I only removed maybe half a dozen files that seemed  wrong (too short to be an encoding of the text).
There certainly is some noise, and the samples are short (about 10.000 characters). I checked these rough entropy values with those on your site, and they do look comparable, but of course they should be taken as purely indicative (values from your site on the left):

If it's any help, what I've been doing as preprocessing before doing PCA is to downcase (this only applies to Roman, Greek, and Cyrillic alphabets AFAIK), treat non-alphabetic characters are spaces, and treat multiple spaces as single spaces.   After that, I try to identify groups of characters which act as phonetic units (for example, gl, tt, and a, but not st because it's two sounds, in Italian).   To do this, you need some idea of how to pronounce the language.   For non-alphabetic scripts, I either find some romanized text or give up.