11-09-2025, 05:52 PM
(This post was last modified: 11-09-2025, 05:54 PM by Jorge_Stolfi.)
(11-09-2025, 04:30 PM)quimqu wrote: You are not allowed to view links. Register or Login to view.Ready to try it! But maybe first I should get a text that has a similar distribution of word lengrhs.
Would You are not allowed to view links. Register or Login to view. do?
It is the Vietnamese translation of the Pentateuch. The file main.wds has one word per line. Lines that start with "a " are words; lines that start with "p " are punctuation, and in particular "p =" is the end of a parag. Other lines may be ignored. The files are described in more detail in the "#"-comments at the beginning. In particular:
# # This file uses a standard Vietnamese encoding in ASCII, VIQR (see
# # table at end of this file) -- except that some accents were
# # remapped to avoid confusion with puctuation ("?"->"@{ß}", "("->"@{µ}",
# # "."->"@{°}"). See the table at the end of this file and the script
# # "fix-encoding".
The encoding is described in detail in the file "You are not allowed to view links. Register or Login to view.". It was the standard way to write e-mail in Vietnamese before Unicode. There are a few hyphenated two-syllable compounds, but most words should be one syllable long.
I also have samples in Mandarin Chinese, but they are in the logographic script, so they are useless for character-level statistics. One should use a romanized transcription like pinyin instead. The files are also in an old pre-Unicode encoding (GB) and conversion to Unicode will not be exactly trivial.
All the best, --jorge

