reading [y] as a null character

[geoffreycaveney]

Hmm yes, it might work like that, it's worth a shot. I should have some time tomorrow to play around with this.
It's not always easy to predict though. For example, why is merging [ch] recommended for entropy optimization but merging [dy] isn't? Nor is [ed], for example. That's why I was planning to test all characters as nulls. It would be interesting if one glyph or a class of glyphs stands out.

Thanks Koen. Let us know the results, I'm looking forward to seeing them! 

As for [ch] vs. [dy] or [ed], sure [d] occurs as part of [dy] in over half of its occurrences, and as part of [ed] in almost half of its occurrences, but there are still thousands of occurrences of [d] without following [y] or preceding [e] -- such as the numerous occurrences of [daiin], et al., for example. 

With [ch], on the other hand, you can almost never find [c] in the ms text without [h] either immediately following or as part of a ligature [c+glyph+h]. So [ch] is rather more clearly a single unit.