28-06-2025, 10:20 PM
(28-06-2025, 09:52 PM)obelus wrote: You are not allowed to view links. Register or Login to view.The unscrambled Cuva sample (continuous text) is shown as grey stars, in satisfactory agreement with your results. They show that the VMS generator is correlated on a scale of 2-4 characters, before the supposed-entropy curve is degraded by its underlying numerical limitations. Any features appearing farther to the right are suspect. We already know that sample length shapes such curves; to make a strong case that encoding method also has a differential effect on them, some kind of sample-length validations would help. How large is your Docta Ignorantia? and how is its length affected by the transformations applied?
Thank you for the explanation, I was a bit struggling to follow the discussion otherwise.
As far as I remember, quimqu was experimenting with GPT models. I think it still could be useful to try longer n-grams with a small GPT model, if the size of the model won't let it memorize all patterns of length n. If it turns out a model can improve its score (get lower perplexity) for high order ngrams despite not having enough weights, it would mean there are some rules or correlations between characters separated by n positions, and the model is able to learn these rules. Does this sound plausible?