This is the word token length distribution of this cipher:
I think we can safely ignore the only word of length 2, since it's at the very end of the ciphertext and could be a truncated token due to the lack of further plaintext.
To me it looks a bit suspiciously narrow for a sum of three independent token lengths, I would expect a longer more gradual right tail. But I don't have a good working understanding of how multiple independent distributions combine into one, so I decided to run a test.
I ran 10 million iterations combining three sets of tokens of length from 0 to 5 with random distributions of lengths within sets and then the same for two sets.
Result for 3 sets (the known distribution is shown as the counts of the actual word tokens, combined distributions are scaled to the same sum of tokens):
Result for 2 sets:
Interestingly, both solutions have one set with a large percentage of the empty token (0 length), about 1/4 for the 2 set solution and 38% for the 3 set solution. 2 set solution has much lower mean squared error. Obviously, it's possible to approximate the distribution perfectly with a 1 set solution.
So, for random combinations of chunks from a few sets, 2 set solutions seem to produce closer length distribution than 3 set solutions, but still it was impossible to find a 2 set solution which would mimic the distribution perfectly.
Now, the actual combinations of tokens for the cipher are not random. So the above doesn't necessarily apply to the plausibility of splitting the tokens into three parts. But I would reexamine the rationale of choosing a three-way split.
I think I will also repeat the experiment with no empty tokens.
I think we can safely ignore the only word of length 2, since it's at the very end of the ciphertext and could be a truncated token due to the lack of further plaintext.
To me it looks a bit suspiciously narrow for a sum of three independent token lengths, I would expect a longer more gradual right tail. But I don't have a good working understanding of how multiple independent distributions combine into one, so I decided to run a test.
I ran 10 million iterations combining three sets of tokens of length from 0 to 5 with random distributions of lengths within sets and then the same for two sets.
Result for 3 sets (the known distribution is shown as the counts of the actual word tokens, combined distributions are scaled to the same sum of tokens):
Code:
Best sets, % of token lengths from 0 to 5: [[3, 37, 33, 12, 5, 7], [3, 14, 40, 38, 0, 2], [38, 17, 41, 0, 0, 1]]
Combined distribution: [ 0 1 8 24 43 50 45 28 15 9 4 0 0 0 0 0]
Known distribution: [ 0 0 0 36 50 57 55 33 2 0 0 0 0 0 0 0]
Error: 42.97Result for 2 sets:
Code:
Best sets, % of token lengths from 0 to 5: [[0, 2, 4, 42, 44, 5], [23, 19, 25, 26, 1, 2]]
Combined distribution: [ 0 1 3 26 47 51 56 33 7 3 0]
Known distribution: [ 0 0 0 36 50 57 55 33 2 0 0]
Error: 16.16Interestingly, both solutions have one set with a large percentage of the empty token (0 length), about 1/4 for the 2 set solution and 38% for the 3 set solution. 2 set solution has much lower mean squared error. Obviously, it's possible to approximate the distribution perfectly with a 1 set solution.
So, for random combinations of chunks from a few sets, 2 set solutions seem to produce closer length distribution than 3 set solutions, but still it was impossible to find a 2 set solution which would mimic the distribution perfectly.
Now, the actual combinations of tokens for the cipher are not random. So the above doesn't necessarily apply to the plausibility of splitting the tokens into three parts. But I would reexamine the rationale of choosing a three-way split.
I think I will also repeat the experiment with no empty tokens.