20-05-2022, 07:24 AM
(19-05-2022, 06:31 PM)MarcoP wrote: You are not allowed to view links. Register or Login to view.I think that treating space as a character is anachronistic.
What do you mean by this, Marco? Does it also mean that they were unlikely to introduce extra spaces?
It feels to me like keeping space as a character in entropy calculations is a way to include word boundaries. E.g. the fact that "y" is often at the end of the word is a useful statistic that can be included.
That said, I agree that removing spaces altogether is the best alternative. But I don't know if removing spaces and then applying transformations (across previous word boundaries) is a valid approach. I haven't tried this yet.

