STEP 1- preprocess
Following Nab’s comments in the last thread, I decided to preprocess the ZL3a-n transcription again using a new custom script. I find the result quite interesting, and I plan to publish all the steps of the process here.
Earlier researchers estimated that there were around 35,000 to 38,000 tokens in previous studies.
In my case, after strict cleaning and tokenization (only [a-z]{1,10} characters, with internal comments removed), I obtain 46,675 tokens and 8,421 unique tokens.
This is not an error as such; it mainly reflects differences in the transcription version (ZL3a-n is more complete), as well as stricter token extraction and the inclusion of all transcribed material, including marginalia and previously excluded elements.
I’ll continue from this clean base to explore distributions and internal structure, following a step-by-step reproducible pipeline.
Following Nab’s comments in the last thread, I decided to preprocess the ZL3a-n transcription again using a new custom script. I find the result quite interesting, and I plan to publish all the steps of the process here.
Earlier researchers estimated that there were around 35,000 to 38,000 tokens in previous studies.
In my case, after strict cleaning and tokenization (only [a-z]{1,10} characters, with internal comments removed), I obtain 46,675 tokens and 8,421 unique tokens.
This is not an error as such; it mainly reflects differences in the transcription version (ZL3a-n is more complete), as well as stricter token extraction and the inclusion of all transcribed material, including marginalia and previously excluded elements.
I’ll continue from this clean base to explore distributions and internal structure, following a step-by-step reproducible pipeline.