“The Library of Babel” by Jorge Luis Borges (1941)

[Jorge_Stolfi]

Another interesting detail is: "[a book] made up of the letters MCV, perversely repeated from the first line to the last" - the most dramatic example of a hopelessly meaningless text that has character conditional entropy equal to zero.

To be pedantic, the average character entropy would be about 20/N bits, where N is the length of the book.  The 20 bits is a guess at the size of a program that would print out that text, on the Librar'y's Official Turing Machine.

All the best, --stolfi

Should'nt Shannon 1st-order entropy be -3*(1/3*ln(1/3)) =~ 1.099, independent from the length of the book?

The Shannon entropy of a stream of characters is how many bits you need, on average, to specify the next character.   IIRC, in Borges story the alphabet was only the 26 capital Latin letters.  It seems reasonable to assume that each of these letters can occur at each position in any book with the same probability, independently of the rest of the text in that book.  Then the Shannon entropy per character (for any order k), will be log_2(26) = ln(26)/ln(2) = ~4.700 bits.

My post above instead used the Kolmogorov information contents of a string, which is the length in bits of the shortest program that outputs the string. 
 
The exact value of the Kolmogorov "entropy" depends on choosing a "machine" which can run those programs, namely a programming language and its semantics.  However, if one believes the Church/Turing hypothesis, the choice only affects the entropy value by a constant factor that depends only one the two machines.

This version of "entropy" captures the intuition that a string of a thousand zeros has less information than a string of 'random' zeros and ones.  It can also handle arbitrarily complicated dependencies between the letters, as long as they are computable; including dependencies that the Shannon kth-order entropy cannot see.   For example, the total Kolmogorov "entropy" of the first N digits of pi is bounded by some relatively small number, independent of N, because they can be generated by an algorithm of fixed size.  On the other hand, almost all strings of N letters have information contents proportional to N, because the shortest program for them will be "print '<string>'"

All the best, --stolfi