@quimqu — the threshold is position ≥ ⌊2L/3⌋, final third of the word by character count. For each BPE segmentation, I count how many of the k−1 boundaries fall at or past that threshold, divided by k−1. Folio mean excludes unsegmented tokens.
So a 9-char token with cuts at {3, 7}: threshold = 6, one qualifies → BC = 0.5. Stars averaging 0.37 means most cuts land in the first two-thirds — right edge is inconsistent across that section. Herbal 0.80+ is the opposite.
Curious how you're approaching boundary detection on your end — different segmentation method, or something positional?
So a 9-char token with cuts at {3, 7}: threshold = 6, one qualifies → BC = 0.5. Stars averaging 0.37 means most cuts land in the first two-thirds — right edge is inconsistent across that section. Herbal 0.80+ is the opposite.
Curious how you're approaching boundary detection on your end — different segmentation method, or something positional?