Opinions on: line as a functional unit

[quimqu]

In the earlier version of the segmentation model, the balance between prefixes and suffixes was off. The algorithm relied on thresholds that were too high, so many true endings were misclassified as “other.” This happened because certain segments, like “dy” or “in,” also appeared inside words, which made the system treat them as ambiguous. To fix this, I relaxed the thresholds and added a small condition on context variety, allowing the model to recognize more genuine suffixes while keeping prefixes stable.

The table shows how these word parts behave depending on their position in a line. Each section corresponds to a different type of line (for instance, the first line of a paragraph or a single-line label), and each row represents a word’s position in that line. The “prefix_start” values show how often words begin with a prefix, while “suffix_end” shows how often they end with a suffix. By comparing these, you can see clear patterns: in paragraphs, prefixes tend to cluster at the start of lines, and suffixes are more common toward the end. This does not occur to the single text lines, but it happens again to labels with more than a word. Curious....