05-09-2025, 12:44 AM
(This post was last modified: 05-09-2025, 12:52 AM by obelus.
Edit Reason: brevity
)
If "perplexity" is in vogue for language models, fine... an earlier thread explained that perplexity is monotonically related to the Shannon entropy. If so, these directionality results are puzzling.
An n-gram entropy can be estimated from the corresponding n-gram frequency distribution in a text sample (whatever the tokenization). When the text is reversed, there will be a one-to-one correspondence between n-grams in the forward and reverse directions (each "abc" forward will be represented by one "cba" in reverse, etc). Therefore any function that can be calculated from n-gram frequencies alone, such as "absolute" entropy, or conditional entropy to any order, must be the same in either direction. How does perplexity break this symmetry?
If the text samples were not globally reversed, character by character as claimed, then statistical asymmetry may be have been introduced when correlations at the word or line level were altered.
An n-gram entropy can be estimated from the corresponding n-gram frequency distribution in a text sample (whatever the tokenization). When the text is reversed, there will be a one-to-one correspondence between n-grams in the forward and reverse directions (each "abc" forward will be represented by one "cba" in reverse, etc). Therefore any function that can be calculated from n-gram frequencies alone, such as "absolute" entropy, or conditional entropy to any order, must be the same in either direction. How does perplexity break this symmetry?
If the text samples were not globally reversed, character by character as claimed, then statistical asymmetry may be have been introduced when correlations at the word or line level were altered.


![[+]](https://www.voynich.ninja/images/collapse_collapsed.png)