Linguistic Patterns Before Decipherment: A Key to Understanding Unknown Texts

[Urtx13]

Earlier researchers estimated that there were around 35,000 to 38,000 tokens in previous studies.
In my case, after strict cleaning and tokenization (only [a-z]{1,10} characters, with internal comments removed), I obtain 46,675 tokens and 8,421 unique tokens.

The ZL transliteration has uncertain word spaces ",", extended EVA "@" codes and illegible characters "?" that are not generally accepted as word separators.

RF1a-n has 38510 words with "," interpreted as a space, 37851 words without.

Nab! Thank you always for your kind feedback.

My preprocessing does not treat commas (,) as word separators, nor does it attempt to recover uncertain or illegible characters (@, ?).

The code strictly extracts only valid [a-z]{1,10} tokens, after removing internal comments (<! ... >) from the lines.

This conservative and reproducible approach means:
-No “guesses” about ambiguous spaces.
-No special handling of uncertain or damaged transcriptions.

As a result, the token count is naturally higher because the ZL3a-n transcription includes more material than earlier versions (such as RF1a-n), and no tokens are artificially excluded unless they fall outside strict [a-z]{1,10} matching.

What do you think?