[split] Verbose cipher?

[nickpelling]

I think that it should (in principle, at least) be possible to write a programme along the following lines:

• Start with a corpus (say, Q20)
  
• Filter out all unreliable words (e g. words containing unreadable glyphs, weirdos, or glyphs that occur less than five times, say)
  
• Create a starting list of tokens based on all the glyphs that remain
  
• Calculate a metric based on the entropy of the text (more on this below)
  
• Find the pair of tokens that, when combined as a new composite token, yields the best metric
  
• Halt if the best metric found is worse than the previous best metric
  
• Else add that new composite token to the list, reparse and repeat
  

The reason this is possible is that adding tokens to the list of tokens shortens the total amount of text but flattens out the stats. So there is a ceiling to the process.

But the trick would be finding a weighted combination of different entropy measures that yield worthwhile results.

Just thought I'd say. :-)