05-08-2017, 02:54 PM
(03-08-2017, 09:25 PM)ReneZ wrote: You are not allowed to view links. Register or Login to view.(03-08-2017, 08:19 PM)Davidsch wrote: You are not allowed to view links. Register or Login to view.No. A 2gram list over a list will have the same entropy as when over the same list if these are the same but expanded 2grams
I'm sorry, I don't follow that, but what I wrote is certainly correct. By expanding individual characters into fixed strings of characters, one reduces the entropy. Essentially by definition.
Not sure that I understand what is discussed here, but in the simple example of duplicating each character the 1st order entropy will remain the same while the 2nd order entropy will change.
Consider a simple sequence 'abababab....' If we modify it to 'aabbaabbaabb....', then neither the alphabet itself nor the frequences of individual characters do not change, hence 1st order entropy is intact. In contrast to that, we obtain new bigrams ('aa' and 'bb' in addition to 'ab' and 'ba'), which changes conditional probablities of characters to appear. In the first sequence, it is all pre-determined: you ever get 'a' after 'b' and 'b' after 'a' (hence h2 = 0). In the second sequence, probabilities to get 'a' after 'b' ('b' after 'a' etc.) reduce to 1/2, and h2 changes to unity.

