15-11-2025, 06:03 PM
(This post was last modified: 15-11-2025, 06:19 PM by quimqu.
Edit Reason: added qokeedy example
)
The Thai experiment has given me a new approach with the model, limiting the one single character segment. Setting a maximum one single character segment to 10%of the segments, has even improved the space detecting KPI in the Voynich:
Length of Voynich text without spaces: 190728
Number of original words (Voynich): 37503
Results on original spaces (Voynich, length aware BPE):
Total word boundaries (start+end): 75006
Correct word starts: 32782 (87.412 percent) *
Correct word ends: 32782 (87.412 percent) *
Words with both boundaries correct: 28627 (76.333 percent)
Boundary precision: 59.192 percent Boundary recall: 87.411 percent Boundary F1: 70.586 percent **
Oversegmentation factor: 1.477 Segments per word: 1.48
Percentage of 1 character segments: 9.98 percent
* the same number is normal, as a space is always at the beginning of a word or at the end of it.
** Boundary precision (59.19%) means that about six in ten predicted boundaries match real spaces, while the rest are internal morphemic splits inside words (for example fachys becomes fa | chy | s). Boundary recall (87.41%) means the model successfully recovers 87% real word boundaries in the text. Boundary F1 (70.59%) summarizes both effects, showing that the model captures most true boundaries while still adding some internal segment breaks.
Example, first 300 characters segmented:
f | a | chys | yk | al | ar | a | taiin | shol | shor | y | cth | r | es | yk | or | shol | dy | | sor | y | ckh | ar | or | yk | air | cht | aiin | shar | as | e | cth | ar | cth | ar | d | an | sy | aiir | sheky | or | ykaiin | shod | cth | o | ary | cth | es | dar | aiin | s | ys | oiin | oteey | ot | eos | rol | oty | cth | i | ar | daiin | okaiin | or | ok | an | sair | yche | ar | cth | aiin | cph | ar | cfh | aiin | ydar | ai | shy | | odar | c' | y | shol | cph | o | yo | ydar | sh | s | cfh | oaiin | sh | odar | yy | shey | shody | ok | cho | yot | chol | chocthy | os | chyd | ain | chor | k | os
In my previous run, qokeedy was splitted in quite different ways ('qok'+'eedy', 'eedy'+'qok' (where two qokeedy are together)). In this run, qokedy and qokeedy appear as full "words" and qokeeedy appears as the junction of two morphems:
Number of occurrences of 'qokedy': 269
Distinct segmentation patterns for 'qokedy': 1
Patterns and counts (most frequent first):
269 times: qokedy
Number of occurrences of 'qokeedy': 303
Distinct segmentation patterns for 'qokeedy': 1
Patterns and counts (most frequent first):
303 times: qokeedy
Number of occurrences of 'qokeeedy': 5
Distinct segmentation patterns for 'qokeeedy': 1
Patterns and counts (most frequent first):
5 times: qokee | edy
Length of Voynich text without spaces: 190728
Number of original words (Voynich): 37503
Results on original spaces (Voynich, length aware BPE):
Total word boundaries (start+end): 75006
Correct word starts: 32782 (87.412 percent) *
Correct word ends: 32782 (87.412 percent) *
Words with both boundaries correct: 28627 (76.333 percent)
Boundary precision: 59.192 percent Boundary recall: 87.411 percent Boundary F1: 70.586 percent **
Oversegmentation factor: 1.477 Segments per word: 1.48
Percentage of 1 character segments: 9.98 percent
* the same number is normal, as a space is always at the beginning of a word or at the end of it.
** Boundary precision (59.19%) means that about six in ten predicted boundaries match real spaces, while the rest are internal morphemic splits inside words (for example fachys becomes fa | chy | s). Boundary recall (87.41%) means the model successfully recovers 87% real word boundaries in the text. Boundary F1 (70.59%) summarizes both effects, showing that the model captures most true boundaries while still adding some internal segment breaks.
Example, first 300 characters segmented:
f | a | chys | yk | al | ar | a | taiin | shol | shor | y | cth | r | es | yk | or | shol | dy | | sor | y | ckh | ar | or | yk | air | cht | aiin | shar | as | e | cth | ar | cth | ar | d | an | sy | aiir | sheky | or | ykaiin | shod | cth | o | ary | cth | es | dar | aiin | s | ys | oiin | oteey | ot | eos | rol | oty | cth | i | ar | daiin | okaiin | or | ok | an | sair | yche | ar | cth | aiin | cph | ar | cfh | aiin | ydar | ai | shy | | odar | c' | y | shol | cph | o | yo | ydar | sh | s | cfh | oaiin | sh | odar | yy | shey | shody | ok | cho | yot | chol | chocthy | os | chyd | ain | chor | k | os
In my previous run, qokeedy was splitted in quite different ways ('qok'+'eedy', 'eedy'+'qok' (where two qokeedy are together)). In this run, qokedy and qokeedy appear as full "words" and qokeeedy appears as the junction of two morphems:
Number of occurrences of 'qokedy': 269
Distinct segmentation patterns for 'qokedy': 1
Patterns and counts (most frequent first):
269 times: qokedy
Number of occurrences of 'qokeedy': 303
Distinct segmentation patterns for 'qokeedy': 1
Patterns and counts (most frequent first):
303 times: qokeedy
Number of occurrences of 'qokeeedy': 5
Distinct segmentation patterns for 'qokeeedy': 1
Patterns and counts (most frequent first):
5 times: qokee | edy