@ oeesordy
The 41.5% is the share of tokens at position 1 in each line that contain at least one internal bigram with negative PMI. So of roughly 3,800 first-position tokens about 41.5% contain such an unusual combination.
Honestly, I can't derive a specific cipher from this yet. The effect is real, but what it means for the underlying system is still open.
Not a null — a null would be content-free, but the first position clearly correlates with properties of the rest of the line (length, token structure, etc.), so it carries information of some kind.
Could you clarify your other points? I'm not sure I fully understand the questions.
The 41.5% is the share of tokens at position 1 in each line that contain at least one internal bigram with negative PMI. So of roughly 3,800 first-position tokens about 41.5% contain such an unusual combination.
Honestly, I can't derive a specific cipher from this yet. The effect is real, but what it means for the underlying system is still open.
Not a null — a null would be content-free, but the first position clearly correlates with properties of the rest of the line (length, token structure, etc.), so it carries information of some kind.
Could you clarify your other points? I'm not sure I fully understand the questions.