(12-05-2026, 08:02 AM)Jorge_Stolfi wrote: You are not allowed to view links. Register or Login to view.In conclusion: if you must do statistical analysis of the text, don't focus on characters, focus on words. And if you need to merge words into word classes (say, to reduce the volume of the results or the sampling noise), try to use classes that are defined by some semantic criterion (like co-occurrence), not by the characters that occur in them (like "qo-words" or "words with gallows").
Well - if only we knew that the word boundaries in the VMS are actually word boundaries. You can cover up to 82 percent of all spaces with just 8 simple rules (I’ll post those rules later in another thread) - which really goes against the idea that they’re normal word boundaries....