[split] Verbose cipher?

[MarcoP]

I think that treating space as a character is anachronistic.  

What do you mean by this, Marco? Does it also mean that they were unlikely to introduce extra spaces?

Hi Koen,
I'll try to clarify, but please remember that I don't know much about cryptography, so it is possible I am totally wrong.

What I mean is that we see space as a character, since it's just one of the keys on our keyboards. This was not the case for medieval scribes.

For instance, this simple substitution does not look plausibly "medieval" to me (here _ represents space):

Code:

plain-text: THIS_IS_UNLIKELY cipherText: ar_qo_qokey_alys

here plain-text _ is cyphered as 'o', and plain-text 'I' is ciphered as '_'.

Does it also mean that they were unlikely to introduce extra spaces?

Spaces were removed in early XV century diplomatic cyphers and extra spaces where used in late XV Century ciphers (e.g. Trithemius), so I think that null spaces could be more acceptable.
I don't think we can exclude anything, but my personal ranking of decreasing preference is:

• Spaces represent actual spaces; one of the reasons is that average word length is approximately correct, so the frequency of spaces is comparable with that of a "normal" plain-text. Of course, this assumption has a number of problems for a cipher or phonetic writing system (e.g. the binomial distribution of word-type lengths). This idea is compatible with options like an artificial language or a cipher based on a nomenclator.
  
• Spaces are nulls; here a problem could be that null spaces are not documented before the end of the XV Century (not totally sure).
  
• The text is a cipher and spaces carry some information that is different from the usual separation of words (see example above).
  

It feels to me like keeping space as a character in entropy calculations is a way to include word boundaries. E.g. the fact that "y" is often at the end of the word is a useful statistic that can be included. 

I see no problem in keeping space in entropy calculations, as long as it is done consistently. This certainly is an important variable for entropy measures. The other two options are removing all spaces or only considering bigrams that are entirely contained "inside" words (i.e. excluding bigrams that contain space). All these options are legitimate, in my opinion.