29-04-2025, 11:49 PM
STEP 1.2
After feedback, I’ve revised the preprocessing step to more accurately reflect the structure and meaning of the ZL3a transcription. This new pipeline fixes several issues that could have introduced artificial artifacts into token-based analyses like TF-IDF or topic modeling.
Key Fixes Implemented
1. Special ASCII codes like @123;
These represent single glyphs and are not actual words.
Fix: They are now completely ignored during tokenization.
2. Alternative readings like [cth:oto]
These indicate transcription uncertainty between multiple candidates.
Fix: Only the first token (e.g., cth) is retained; the rest is ignored.
3. Tokens with ? characters
These represent uncertain or corrupted readings.
Fix: Entire tokens containing ? are excluded from the vocabulary.
4. Inline comments like <!...>
These were already being filtered out, and continue to be excluded correctly.
Resulting Token Statistics (after filtering):
-Total number of tokens (cleaned EVA): 45,037
-Unique tokens in final vocabulary: 8,542
After feedback, I’ve revised the preprocessing step to more accurately reflect the structure and meaning of the ZL3a transcription. This new pipeline fixes several issues that could have introduced artificial artifacts into token-based analyses like TF-IDF or topic modeling.
Key Fixes Implemented
1. Special ASCII codes like @123;
These represent single glyphs and are not actual words.
Fix: They are now completely ignored during tokenization.
2. Alternative readings like [cth:oto]
These indicate transcription uncertainty between multiple candidates.
Fix: Only the first token (e.g., cth) is retained; the rest is ignored.
3. Tokens with ? characters
These represent uncertain or corrupted readings.
Fix: Entire tokens containing ? are excluded from the vocabulary.
4. Inline comments like <!...>
These were already being filtered out, and continue to be excluded correctly.
Resulting Token Statistics (after filtering):
-Total number of tokens (cleaned EVA): 45,037
-Unique tokens in final vocabulary: 8,542