Least effort generator

[tikonen]

This is certainly a possibility, the idea itself is not particularly new. Did you look into a way to generate statistically probable distributions or sequences of these tokens?

Also, there can be easy counterexamples. If you take Chinese written phonetically (pinyin), then overwhelmingly all words will be created using a combination of 1-3 simple tokens. It's likely the same is true for Vietnamese, which actually uses a script based on Latin characters. This doesn't mean that all Vietnamese books are just a few scribes randomly combining tokens.

I'm planning to map the sequences to numeric form for easier analysis. In any case if this kind of system was used as a training tool it's unclear how it would show in the final text. Maybe nothing conclusive can be said but as said earlier I would like to come up with a simple system that could have been used with the knowledge available at the time.

Thank you for pointing out some counter examples, I've no knowledge of Pinyin and will have to look into it as a case example. Based on several text examples I glanced I don't immediately see obvious comparison to VM style text. Of course Vietnamese nor South-East Asian books cannot be generated with a random method but my assumption stands that VM is just gibberish with just enough structure to make it look like a real language. Even today many comedians are really skilled in talking in gibberish that sounds exactly like a real language when they or listeners don't speak the language.