17-11-2025, 02:50 PM
I have the following versions of the Pentateuch (first 5 book of the Old Testament)
translated to the follwoing languages:
You are not allowed to view links. Register or Login to view. Mandarin (Union)
You are not allowed to view links. Register or Login to view. Mandarin (another transl.)
You are not allowed to view links. Register or Login to view. Hebrew
You are not allowed to view links. Register or Login to view. Latin (Vulgate)
You are not allowed to view links. Register or Login to view. Russian (Synodal)
You are not allowed to view links. Register or Login to view. Vietnamese (Cadaman)
Those texts have been cleaned, and those files have one token per line,
with each token clearly marked as word, punctuation, or symbol. However
the character set and spelling in each file is specific to each
language, so it may need some re-coding before it can be fed to an
analysis program.
The Chinese texts, in particular, use an old encoding with two bytes per
Chinese character. There is a Linux program "autogb -i gb -o utf8" that
converts them to the modern Unicode/UTF8.
For character and word structure analysys the Chinese characters would
then have to be converted to the phonetic pinyin Romanization; but for
word-level analysis (like those graph measures) it would be enough to
map each Chinese Unicode character to some distinct ascii string, like
'U+FCBA' or '{40567}''
Hope it helps, --stolfi
translated to the follwoing languages:
You are not allowed to view links. Register or Login to view. Mandarin (Union)
You are not allowed to view links. Register or Login to view. Mandarin (another transl.)
You are not allowed to view links. Register or Login to view. Hebrew
You are not allowed to view links. Register or Login to view. Latin (Vulgate)
You are not allowed to view links. Register or Login to view. Russian (Synodal)
You are not allowed to view links. Register or Login to view. Vietnamese (Cadaman)
Those texts have been cleaned, and those files have one token per line,
with each token clearly marked as word, punctuation, or symbol. However
the character set and spelling in each file is specific to each
language, so it may need some re-coding before it can be fed to an
analysis program.
The Chinese texts, in particular, use an old encoding with two bytes per
Chinese character. There is a Linux program "autogb -i gb -o utf8" that
converts them to the modern Unicode/UTF8.
For character and word structure analysys the Chinese characters would
then have to be converted to the phonetic pinyin Romanization; but for
word-level analysis (like those graph measures) it would be enough to
map each Chinese Unicode character to some distinct ascii string, like
'U+FCBA' or '{40567}''
Hope it helps, --stolfi

