04-04-2018, 07:46 PM
I use 'valid' for two reasons:
1) I can't be sure that a rare word is correct/true/real; and
2) I can't be sure that a rare word is representative of the text.
For point 1), we know that there are transcription errors. That situation may improve over time, but there's a real possibility that words with one token have been misread. Likewise, there is an unknowable number of writing errors. We can never identify them until we can read the text.
The answer is to build a wall around the problem. We can guess that the same reading and writing errors haven't been made many times, thus the more common a word is the less likely that it is incorrect. By putting in place an (admittedly) arbitrary cut-off we avoid basing our results on false words.
Of course, that wall will also enclose a number of real words, which is where point 2) comes in. Some words which are uncommon or rare aren't representative of the text as a whole even if their readings are correct. They contain combinations of characters, such as [arl], or chimeric structures, like [qokaiinos], which are themselves rare.
I don't know why some words are unlike the rest. They should be studied and may even yield a lot of information about the text. But rules which attempt to include them would stretch so wide as to become useless. My goal is not to describe the text in detail but to typify it into patterns.
1) I can't be sure that a rare word is correct/true/real; and
2) I can't be sure that a rare word is representative of the text.
For point 1), we know that there are transcription errors. That situation may improve over time, but there's a real possibility that words with one token have been misread. Likewise, there is an unknowable number of writing errors. We can never identify them until we can read the text.
The answer is to build a wall around the problem. We can guess that the same reading and writing errors haven't been made many times, thus the more common a word is the less likely that it is incorrect. By putting in place an (admittedly) arbitrary cut-off we avoid basing our results on false words.
Of course, that wall will also enclose a number of real words, which is where point 2) comes in. Some words which are uncommon or rare aren't representative of the text as a whole even if their readings are correct. They contain combinations of characters, such as [arl], or chimeric structures, like [qokaiinos], which are themselves rare.
I don't know why some words are unlike the rest. They should be studied and may even yield a lot of information about the text. But rules which attempt to include them would stretch so wide as to become useless. My goal is not to describe the text in detail but to typify it into patterns.