Quote:Julian:
I think the statistics will look very different for an n-gram substitution scheme with nulls, as opposed to a simple substitution scheme which is clearly discounted.
It certainly would. The difficulty is in figuring out which ones might be nulls (I've been struggling with this for years).
As soon as you postulate nulls, you have to assume the spaces (or some of the spaces) might also be nulls.
And... as soon as you try interpreting the spaces in different ways, Capelli goes out the window, because a high proportion of the patterns in the VMS text that behave in ways similar to Latin abbreviations (or appear to) are at the beginnings and ends of tokens. Re-interpreting spaces changes the whole expansion process.