[Blog Post] A Universal Template

[Emma May Smith]

But if there is a very plausible missing word, then why is this word missing in the first place? There must have been something working against its inclusion.

Let's say we have the ability to calculate the "plausibility" of a word (say, as the the product of bigram likelihoods). Then the plausibility for all possible strings could be calculated and checked against the text. This would provide us with a list of most plausible missing words.

The test then would be to discover if a) they're distributed randomly across all plausible words (which would be natural language like), or b) are governed by further rules, such as longer range dependencies (hypothetically, no word should contain two instances of [l] or have [k] exactly two characters after [d]). These further rules could be plugged back into the plausibility calculation, and the list of plausible missing word list rechecked. If done iteratively we should be left either with a situation where all the missing words are highly implausible or randomly spread across all plausible words.

Would something like this exhaust the word formation rules?

The same "plausibility index" could likewise be checked against not type presence but token amount, with a token/plausibility ratio. This might even better give a sense of the "naturalness" or not of the text. Is the ratio constant? Is [daiin] simply the most plausible word rather than the most common one?