Something that I very much want to do (I will have time next week) and was prompted by Torsten Timm's post here: You are not allowed to view links. Register or Login to view.
How abnormal are paragraphs/pages/quires? Does homogeneity improve at the end? All pages of Q13 and Q20 look normal to me relative to their quire, without all the weird spikes in frequencies seen in the early herbal pages, but this should be checked.
I need statistics / probability calculation advice.
What would be the best way to "objectively" (for a given transliteration) measure how normal (or abnormal) a chunk of text is compared to its context (page / quire / entire Currier language)?
I guess it would be best to count mono-/bi-/tri-grams and compare (how?) these statistics to the expected binomial distribution. Of course large variations in frequencies are expected when the size of the sample is small, so the metric has to take the size of the sample into account somehow.
Or maybe the probability of the sample (paragraph or page) could be simply calculated (assuming independence between the probabilities of each n-gram, not true actually but maybe good enough for a quick approximation), the same way as in the Hill climbing algorithm: multiply probabilities (or better add logarithms of probabilities) of presence of each n-gram estimated from overall frequency, then normalize for size (divide the sum by the size of the sample)?
Quote:If it comes to statistics for the VMS it is important to keep in mind that the text isn't homogenous. The text differs from Currier A to Currier B, from quire to quire, from bifolio to bifolio, from page to page and even within a paragraph or line.
How abnormal are paragraphs/pages/quires? Does homogeneity improve at the end? All pages of Q13 and Q20 look normal to me relative to their quire, without all the weird spikes in frequencies seen in the early herbal pages, but this should be checked.
I need statistics / probability calculation advice.
What would be the best way to "objectively" (for a given transliteration) measure how normal (or abnormal) a chunk of text is compared to its context (page / quire / entire Currier language)?
I guess it would be best to count mono-/bi-/tri-grams and compare (how?) these statistics to the expected binomial distribution. Of course large variations in frequencies are expected when the size of the sample is small, so the metric has to take the size of the sample into account somehow.
Or maybe the probability of the sample (paragraph or page) could be simply calculated (assuming independence between the probabilities of each n-gram, not true actually but maybe good enough for a quick approximation), the same way as in the Hill climbing algorithm: multiply probabilities (or better add logarithms of probabilities) of presence of each n-gram estimated from overall frequency, then normalize for size (divide the sum by the size of the sample)?
![[+]](https://www.voynich.ninja/images/collapse_collapsed.png)