25-06-2025, 11:46 AM
(25-06-2025, 10:18 AM)Koen G Wrote: You are not allowed to view links. Register or Login to view.Ah, I see. So you have a sliding window of n, and then you see how well you can predict the next n characters? I just can't intuitively grasp what this tells us. It's too abstract - a measure of the information density of the writing system.
Actually, maybe it was me who misunderstood what quimqu was computing. According to the intro post:
Quote:An n-gram model learns the likelihood of seeing a particular character given the previous n-1 characters.
So, for 5-gram it should be guessing the next character based on the previous 4 characters.
For example, given qoke*, what is *?
But then it's strange that the perplexity goes up with longer ngrams, it shouldn't.
So, I don't understand what is going on here.