13-03-2025, 02:54 PM
(This post was last modified: 13-03-2025, 03:04 PM by zachary.kaelan.)
Spaces do appear to have some significance as a separator, but based on the large number of long, unique vords that appear to be composed of other vords, and the number of ambiguous spaces, it may be useful to do analysis without spaces (still separating by lines), and see if any patterns emerge that line up with the use of spaces.
One of the biggest problems of variable length n-grams is telling things like if “okeol” is “ok-eol” or “oke-ol”. Spaces could serve as just a way of separating out ambiguous n-grams.
One of the biggest problems of variable length n-grams is telling things like if “okeol” is “ok-eol” or “oke-ol”. Spaces could serve as just a way of separating out ambiguous n-grams.