20-12-2021, 05:50 PM
Marco and Koen, one potential solution I see to this paradox is to identify ngrams that would make good candidates for metadata, and ignoring these whilst looking for ngram patterns across lines and down paragraphs and pages. The finer points of the experimental methodology I'd want to spend some time crafting include:
If this went well, I would then want to compare the plots of ngram occurrence by horizontal and vertical position in the VMs, with those of English texts that clearly do not contain embedded metadata. I'd be especially curious to see a comparison to an English text that was formatted to insert a line break after every sentence, continue no sentences over more than one line, and continue no paragraphs over more than one page.
- A better-argued justification, with reference to mainstream history and information science, for why the VMs text containing embedded metadata is likely enough to be worth taking seriously as a hypothesis.
- A point-system rubric for vetting candidate ngrams on their likelihood of functioning as metadata, if there are any that do in the first place.
- Specifying and justifying a cutoff score, and marking those vords that exceed it for non-consideration in statistical analyses. This could be done in several different ways, each with its own altered copy of a machine-readable VMs transcription. For example, one copy could replace each glyph of suspected metadata with EVA=w, while another could delete them as if they were not there at all. One version could replace or delete candidate metadata string wherever they occur in the text, while another could leave ngrams unchanged if they don't appear to occur at any obvious transition point.
- Feed these stripped VMs transcriptions to Patrick and Obelus's bots, and compare the outputs to those of the virgin VMs transcription.
If this went well, I would then want to compare the plots of ngram occurrence by horizontal and vertical position in the VMs, with those of English texts that clearly do not contain embedded metadata. I'd be especially curious to see a comparison to an English text that was formatted to insert a line break after every sentence, continue no sentences over more than one line, and continue no paragraphs over more than one page.

