Opinions on: line as a functional unit

[Jorge_Stolfi]

You are not allowed to view links. Register or Login to view.

Wow, Patrick's paper is quite a big meal to digest.

Some things I wish he had done differently or deeper: 

1. Remove the head lines of parags.  Among other things, that line is likely to have special contents (like plant names and aliases), which could well imply different word frequencies and positional patterns, and hence the same for characters and digraphs.

One problem to watch for here is that parag breaks are sometimes not obvious.  In the Stars section, in particular, I suspect that there there is a run of 5-6 parags that were joined by the Scribe (a newbie?) into a single parag, before he returned to the normal format.

2. Limit the analysis to just one of the sections with substantial running text -- Herbal-A, Herbal-B, Bio, and Stars.  If the anomalies are real, they should be noticeable, and probably even stronger, in one of those sections.   If they turn out to be absent or different in other sections, that by itself would be important information.

3. Try to identify the words that are responsible for the anomalies.  Maybe I have misread the tables in the paper, but among the Sh/Ch word pairs, some seem to have greater positional bias than others. 

Could it be, for example, that the most leftward member of the pair often occurs after a long word, while the other member more often occurs after a short one? That could perhaps explain the positional anomaly as a consequence of the line-breaking word-length bias.

Or maybe the two members of the pair can get fused or split at different rates in the transcription.  So that that some of the Sheols are actually Sheoldy while most Cheols are indeed Cheols.  I can't think how this possible confounding factor could be addressed.  Although this may be one case 

All the best, --stolfi