Elephant in the Room Solution Considerations

[Jorge_Stolfi]

Currier is better because it did take into account the visual unity of double c’s (with or without diacritic), and I think is much better than the EVA. EVA is NOT a reliable transcription. V101 is the best of them all, so far. They can be used to perhaps come up with a more visually representing transcription. I would rather use c-c or c-^-c for example, or c->-c, or c-°-c, depending on the kind of diacritics used or not, for instance.

That assessment depends on two specific assumptions: (1) that the Author was the same person as the Scribe, and (2) that the alphabet he designed distinguished glyphs by small changes in their shape.  (And also on other assumptions that I am not allowed to mention here.)

By Author I mean the person who devised the script, chose the contents, and determined the sequence of words on each page.  

By Scribe I mean the person who actually put the ink on the vellum to produce the VMS we have now. 

Without those two assumptions, one must assume that the Scribe would not be able to reproduce the precise shapes of the glyphs, so those fine details are just noise with no significant information about the contents of  the book.  And, moreover, that normal random variations of handwriting can deform glyphs so much that they look like other glyphs. 

But then "splitter" encodings that take into account small changes in glyph shape, like VT101, are bad encodings, because they require the transcriber to record that noise and make more arbitrary decisions.  With EVA the transcriber often has to arbitrarily decide whether an ambiguous glyph is an r or an s. With a splitter encoding he must make several such decisions on every glyph, about the length and shape and angle of body and plume and tail... And such encodings are arbitrary anyway: why distinguish only six types of diacritics on a Ch, and not seventeen?

A splitter encoding can also make analysis more cumbersome and less revealing. With EVA, the most common word in the book is daiin, and we can compare its frequency from section to section.  With a splitter encoding that word may become five or six different words, and mere handwriting variations may radically change their frequencies from section to section.

If you do believe in (1) and (2) above, a "splitter" encoding like VT101 can make sense.  If you don't believe (1) and (2), EVA is much better.

In fact, EVA may be too "splitting" itself.  For some analyses, it may be a good idea to first map a and o to the same letter, and r and s, and maybe even Ch and ee.  That may lose some information, but will reduce the noise that comes from handwriting variation and transcriber errors.

As an analogy, consider doing linguistic analysis of English text.  It is usually better to first map the whole text to lowercase and remove any italics and boldface markup.  This pre-processing loses some significant information, such as the difference between the proper name "White" and the common word "white", and the relative emphasis of some words.  But it also removes the noise that comes from capitalization of the first word of sentences; so that, in the sentence"The man sees the dog", words 1 and 4 can be recognized as being the same word.

All the best, --stolfi