Agreeing on standard transliteration files to use

[ReneZ]

I find this discussion a bit surprising, because all that is being discussed exists already exists since quite a while.

Let's take Nick's most recent post as an example:

I think that it should (in principle, at least) be possible to write a programme along the following lines:

• Start with a corpus (say, Q20)
  
  
• Filter out all unreliable words (e g. words containing unreadable glyphs, weirdos, or glyphs that occur less than five times, say)
  
  
• Create a starting list of tokens based on all the glyphs that remain
  
  
• Calculate a metric based on the entropy of the text (more on this below)
  
  
• Find the pair of tokens that, when combined as a new composite token, yields the best metric
  
  
• Halt if the best metric found is worse than the previous best metric
  
  
• Else add that new composite token to the list, reparse and repeat
  
  

The reason this is possible is that adding tokens to the list of tokens shortens the total amount of text but flattens out the stats. So there is a ceiling to the process.

Suppose that one wants to do this not just once, but for different cases:
- the whole MS
- the A language corpus
- quire 20 or quire 13
- and for the v101 transliteration as well
- and also when converted to Cuva

then one would be spending a lot of time just for preparing manually each version of the text to be processed. (The above bullets imply 16 different text versions, but it is still just a selection).

For me, all of the first steps in Nick's list and all of the bullets I listed after the quote are simple commands and can be combined in a single script, due to the use of some standards.

The only 'work' that remains is the interesting part: computing a clever metric and a clever optimisation scheme.