Of bookshelves and statistics

[Mauro]

Q20 is the interesting one, so I went looking for the hidden variable. I think I found it:

• Language B has two ways of writing. The herbal B pages use words like daiin, aiin, otaiin. The biological pages (Q13) use qokeey, qokeedy, shedy, and repeat them a lot.
  
• Q20 mixes both, sheet by sheet. Sheet 105-114 is written almost like herbal B, sheets 103-116 and 108-111 almost like the biological section, the others in between. Each sheet keeps its style from the first line to the last.
  
• I trained word models on the rest of the manuscript, without Q20, and scored each Q20 page. “How biological is this page” follows npcompl33t’s order at 0.93 (1 = perfect).
  
• It isn’t page length. With every page cut to the same number of words, nothing changes.
  

That’s why Layfield and Davis’s LSA and npcompl33t’s curve give the same Q20 order: both measure which words a page uses, so both see the same thing. Height measured with two rulers.

Or height and weight, the latter measured on a scale. Yes this is possible and looks reasonable to me (but see the last sentence of this post)

Anyway thank you @quimqu, a lot of interesting explanations. So regarding Q20, as I understand it at the moment: there are indeed two gradients in Q20, the frequency of -aiin words and the frequency of -eey words, and they co-vary, one opposite the other, so ordering along one frequency orders also along the second one. The problem is the two frequency variables are not independent: the frequency of -aiin words is per force inversely proportional to the frequency of -eey word.

One glimmer of hope I see is that the proportionality does not need to be perfect: with many words which are neither aiin-type nor eey-type there's some margin for -aiin and -eey to have some independence, so maybe (just a wild guess) a signal could be extracted from there. You AI-assisted guys look like wizards to me, so who knows 

I'd put more hopes in finding a gradient in a relatively (but not too much) rare textual marker which is shown to co-vary with -aiin and -eey, but being rare should not be affected that much by the rule that all frequencies must add up to one (another wild guess). Or, as you say, we would need extra-textual clues, which are hard to come by (even if iirc @Stolfi has some opinions on the sequence the nymphs were drawn).

I'd like also to hear @npcompl33t's opinon (and of course Lisa's, that would be top, but I don't pretend that much!). And from everyone else obviously.