- Options
- Login
- Desktop Style
Options
Word repetition analysis of the different hands
- Word repetition analysis of the different hands
-
Word repetition analysis of the different hands
anejati > 3 hours ago
I haven't seen this analysis posted here before. It's been suggested that repeating word sequences (like sheol sheol sheol) are You are not allowed to view links. Register or Login to view. caused by scribal error. If this were the case, you would expect there to be differences amongst the different scribal hands, reflecting different levels of experience, scribal accuracy, etc. So to test this, I wrote some code to analyze the variations of word repetition among the 5 different known hands. In summary, these are the results (from the RF EVA transcription given You are not allowed to view links. Register or Login to view.):
Code:Hand | Total Words | 2-Word Repetitions | 3-Word Repetitions
------------------------------------------------------------------------
1 | 8,626 | 64 | 1
2 | 8,947 | 60 | 2
3 | 11,547 | 62 | 1
4 | 654 | 2 | 0
5 | 866 | 1 | 0
I then did a Bayesian analysis, modeling the repetition probability of each hand as a Beta distribution and calculating the likelihood that the different hands actually represent different repetition rates. The hands with more words (1, 2, 3) lead to a narrower, 'tighter' distribution because we have more data. The low-resource hands (4 & 5) have broader distributions (the repetition rates are consistent with a larger range of underlying repetition probabilities). For 3-word repetitions, there isn't enough data to draw meaningful conclusions except to say that there's nothing to indicate any statistically significant differences. For 2-word repetitions, here are the results.
In summary: Mostly, hands have repetition rates that are consistent with each other. The largest statistically meaningful difference seems to be between hand 3 and hand 1. The Hand 1 mean rate is 0.753%. The hand 3 mean rate is 0.546%. The probability that Hand 1 > Hand 3 is 96.6% according to this model. Interestingly, hand 4 seems consistent with all the other hands (except 1) despite the fact that hand 4 seems to mostly write in "labelese" and not "prose", indicating that the repetitions may actually be a feature of the language rather than a mistake.
I also ran this analysis on the version of the EVA transcription by Lisa Fagin Davis where some of the gallows characters are taken to be substitutions of each other. The results are broadly similar, and I get around 84% likelihood of hand 1 > hand 3
There are some limitations here due to limitations of my script, for example I haven't counted word repetitions that cross line boundaries. But this seems unlikely to change the result.
What are my conclusions? Well it's no surprise when it comes to the VMS but it's hard to conclude anything. The consistency of repetitions amongst the hands seems to indicate that the repetitions are a language feature, not mistakes. The higher rate of repetitions for hand 1 vs hand 3 could either be due to mistakes or differences in Currier A (which is predominantly what hand 1 writes in). -
RE: Word repetition analysis of the different hands
Jorge_Stolfi > 2 hours ago
(3 hours ago)anejati Wrote: You are not allowed to view links. Register or Login to view.So to test this, I wrote some code to analyze the variations of word repetition among the 5 different known hands. In summary, these are the results (from the RF EVA transcription given You are not allowed to view links. Register or Login to view.):
Code:Hand | Total Words | 2-Word Repetitions | 3-Word Repetitions
------------------------------------------------------------------------
1 | 8,626 | 64 | 1
2 | 8,947 | 60 | 2
3 | 11,547 | 62 | 1
4 | 654 | 2 | 0
5 | 866 | 1 | 0
Thanks for the intriguing and useful results!
However, different "hands" may mean different topics; and it seems that word statistics are largely specific to topic, as one would expect. Even the difference between Herbal-A and Herbal-B may be a consequence of those texts having been taken from two different sources.
Can you see a difference between two hands on the same section (Herbal_A, Herbal-B, Bio, etc..)?
All the best, --stolfi -
RE: Word repetition analysis of the different hands
nablator > 1 hour ago
(3 hours ago)anejati Wrote: You are not allowed to view links. Register or Login to view.So to test this, I wrote some code to analyze the variations of word repetition among the 5 different known hands. In summary, these are the results (from the RF EVA transcription given You are not allowed to view links. Register or Login to view.):
Code:Hand | Total Words | 2-Word Repetitions | 3-Word Repetitions
------------------------------------------------------------------------
1 | 8,626 | 64 | 1
2 | 8,947 | 60 | 2
3 | 11,547 | 62 | 1
4 | 654 | 2 | 0
5 | 866 | 1 | 0
Quick check on reduplications: 267 lines total
You should have a lot more in the "2-Word Repetitions" column.
Quick check on triple words:
For LFD hand 1 I have 2: You are not allowed to view links. Register or Login to view. and f47r: chol chol chol
For LFD hand 2 I have 4:
f40r: okaiin okaiin okaiin
f75r: qokedy qokedy qokedy qokedy
f79v: qokedy qokedy qokedy
f86v3: ytaiin ytaiin ytaiin
For LFD hand 3 I have 2:
f108v: qokeedy qokeedy qokeedy
f104v: sheol sheol sheol
For LFD hand 4 I have 1:
fRos: otody otody otody -
RE: Word repetition analysis of the different hands
anejati > 1 hour ago
I primarily wanted to focus on the question of scribal accuracy. I suspect that analysis of word repetitions based on Currier languages has probably been looked at quite a lot. But it's a worthwhile question to ask if Currier language differences explain most of the discrepancy between hands 1 and 3. The results are interesting.
Language A Mean Repetition Rate: 0.7273%
Language B Mean Repetition Rate: 0.5787%
Confidence that A > B: 93.0%
So based on this it actually seems the confidence level of word repetitions of hand 1 vs hand 3 being different (96.6%) is more significant than word repetitions of lang A vs lang B being different, but not by a lot.
I think the main issue preventing any solid conclusions is that there's a lot of correlation between hands and langs, so separating out the differences is difficult. The most statistically significant differences seem to be between hand 1 and hand 3, which are also the hands that use the most different langs. -
RE: Word repetition analysis of the different hands
anejati > 1 hour ago
(1 hour ago)nablator Wrote: You are not allowed to view links. Register or Login to view.Quick check on reduplications: 267 lines total
You should have a lot more in the "2-Word Repetitions" column.
Quick check on triple words:
For LFD hand 1 I have 2: You are not allowed to view links. Register or Login to view. and f47r: chol chol chol
For LFD hand 2 I have 4:
f40r: okaiin okaiin okaiin
f75r: qokedy qokedy qokedy qokedy
f79v: qokedy qokedy qokedy
f86v3: ytaiin ytaiin ytaiin
For LFD hand 3 I have 2:
f108v: qokeedy qokeedy qokeedy
f104v: sheol sheol sheol
For LFD hand 4 I have 1:
fRos: otody otody otody
What EVA transcription file are you using? -
RE: Word repetition analysis of the different hands
nablator > 1 hour ago
(1 hour ago)anejati Wrote: You are not allowed to view links. Register or Login to view.What EVA transcription file are you using?
RF1a-n or RF1b-er.
egrep "\b(\w+)\.\1\b" RF1b-er.txt | wc -l
267
egrep "\b(\w+)\.\1\.\1\b" RF1b-er.txt | wc -l
10
The one on f81r.5 may or may not be counted depending on how you interpret ",". -
RE: Word repetition analysis of the different hands
anejati > 40 minutes ago
Code:Hand | Total Words | 2-Word Repetitions | 3-Word Repetitions
------------------------------------------------------------------------
1 | 10,633 | 82 | 2
2 | 10,934 | 71 | 4
3 | 11,921 | 64 | 1
4 | 2,938 | 8 | 0
5 | 902 | 1 | 0
And:
Comparison: Hand 1 vs Hand 3
Hand 1 Mean Rate: 0.780%
Hand 3 Mean Rate: 0.545%
Probability that Hand 1 > Hand 3: 98.6%