Is there any protocols to identify what sort of document MS-408 is?
#20
(17-07-2026, 12:46 AM)ReneZ wrote: You are not allowed to view links. Register or Login to view.People usually underestimate how significant a change you need to a text in order to appreciably change the character-level entropy values.

Indeed: deleting, rearranging, and duplicating words will usually have very little effect on character entropy, whether simple (h0) or conditional (h1,h2,..).  To lower the entropy one would have to delete words with high character entropy and replicate words with low entropy many times.  The banana ananas text ananas banana banana would banana become daiin daiin raiiin rather daiin daiin boring daiin kaiin.

But changes in the spelling (or encryption method, or gibberish generator) can easily change any hk to any desired value, from near zero to the maximum of log2(N) for an alphabet of N characters (~4.7 for lowercase Latin letters plus space). If the h0 of the plaintext is (say) 2.0, one can raise it to ~(2.0+4.7)/2 = ~3.3 by inserting a random letter after each original letter or space; or reduce it to ~1.0 by inserting an "s" after each letter or space.  

If the plaintext is all in lower case, one can add 1 to every hk by randomly changing half the letters to upper case.   

One can even raise every hk to the maximum, without changing the length of the text or the alphabet, by encrypting it with a Vigenère cipher with the 26-letter key ABCDE...XYZ.

Conversely, if h0 is greater than h1, one can lower h0 without changing the length of the message by encoding each letter with a simple cipher that depends on the previous letter.  For example, suppose the plaintext strictly alternates between the "consonants" K,T and the "vowels" A, O, and KA,KO,TA,TO  have the same frequency (~25%).  Then h0 will be ~2 bits/letter but h1 will be ~1.  Then one can lower h0 to ~1 by replacing K☛A, T☛O except at the first letter of each word.

All the best, --stolfi
Reply


Messages In This Thread
RE: Is there any protocols to identify what sort of document MS-408 is? - by Jorge_Stolfi - 17-07-2026, 09:01 AM

Possibly Related Threads…
Thread Author Replies Views Last Post
  IMPORTANT ANNOUNCEMENT (well, sort of) Jorge_Stolfi 88 26,529 07-02-2026, 09:03 AM
Last Post: Koen G

Forum Jump:


Users browsing this thread: 1 Guest(s)