Experiments with language corpora

[MarcoP]

Thank you, Marco, excellent work again. There's not much VM research being shared at the moment so it's nice to read something like this.

Given examples like Korean, does this mean that low entropy can be caused - in part or entirely - by the writing system, independent of the language? 

Of course the writing system can have an immense impact on every statistical study. For instance, the very common medieval practice of using different symbols for a few letters if they are word initial/final or inside words affects as basic a quantity as the apparent size of the alphabet.

Also, might the choice of text influence the results? The Declaration, a series of statements about what is or "shall be", excludes certain language forms from occurring. Past tenses, for example. 

These statistics are so "macroscopic" that they are not likely to be affected by the subject of the text. I am fairly sure this is true for entropy. In principle, repetition could be more common in some texts than in others, but when the phenomenon is as frequent as in the VMS (about one occurrence every 100 words) it is likely a basic feature of the language, in my opinion.

I've been looking around a bit for a readily available corpus but haven't found anything useful yet.
On this site they have the Bible in hundreds of languages: You are not allowed to view links. Register or Login to view.
Perhaps with some planning and forum collaboration we could compile a corpus from something like this?

This seems feasible. I would be surprised that nobody had already done this.
As an example, I have added to the plot the complete King James Bible, with pre-processing (punctuation removed and converted to lower case).
These are the numbers compared with the "eng" file from the Human Rights corpus:

Code:

KING_JAMES 0.053 2.992 eng        0.094 2.981

The yellow square in the plot is the complete, pre-processed King James. The blue circle is the un-pre-processed Human Rights "eng" file.