About the construction of lines in the MS

[Jorge_Stolfi]

I suppose you are using a neural network to measure the anomalies, so you cannot tell what exactly it is measuring?

I’m not using a neural net. It’s a simple score. Roughly, I learn what typical line endings look like and what typical line beginnings look like, at the character level. Then for any cut I score how “end-like” the left side is and how “start-like” the right side is, compared to the overall background.

Sorry, I was confused by your use of the word "learn".   What do you mean by that? 

Do mean that for, each character c, you compute 

  F0( c)  = frequency of c as the word-final character for "middle" words
  FV( c) = frequency of c as the line-final character in the actual VMS text
  FT( c) = frequency of c as line-final character after re-justifying with TLA
  FS( c) = frequency of c as line-final character after re-justifying with SLA

then you compute scores mV, mS, mT that compare the distributions FV, FT, FS with F0?  

Or do you use those distributions to build three "line break predictors" for V, T, and S, and then compute scores that compare their performances somehow?

I ask because I wonder if the results you see are being determined by some other specific character, besides m (or n), that is more (or less) frequent in FV but is not created (or suppressed) by SLA.

All the best, --stolfi