Linguistic Patterns Before Decipherment: A Key to Understanding Unknown Texts

[ReneZ]

-Total number of tokens (cleaned EVA): 45,037

Too many, something is wrong.

I agree.

Note that:
- reliable removal of <!..> comments can be done by ivtt
- reliable selection of the first option of [ | ] can be done by ivtt
- reliable removal of words includung ? can be done by ivtt

Not having to write your own code for that means saving time and removing the risk of bugs.

Handling @123; is  bit more tricky. ivtt can convert them to single high-ascii bytes for you, but then you need to include these in your selection of characters. By the way, you can remove 0-9 from your selection of characters.

Alternatively, you can use bitrans to convert them to 'nearest basic Eva'.
Step 1: convert full Eva to STA (can be skipped if you use the STA file in the first place)
Step 2: convert STA to basic Eva (irreversible) using the file "Beva.bit" which is provided

If you start with STA, you can use any of the files, and ZL, GC, RF should give similar statistics.
Some of these statistics are also provided at my web site, Table 13 on You are not allowed to view links. Register or Login to view.