[Article] A Glyph Is Not a Letter, a Token Is Not a Word, a Space Is Not a Space

[pfeaster]

I'm still working my way through the paper, but one section particularly catches my attention, since it touches on a phenomenon I'd brought up previously but that hasn't otherwise been much discussed.

A few years ago, when I was trying to assess how predictably spaces could be reintroduced into Voynichese text from which the spaces had been removed based on rules of glyph adjacency (e.g., always put a space between [nq]), I noticed that the glyph pairs that displayed the most inconsistent spacing tended also to involve the most ambiguous spacing.

For example, according to the ZL transcription I was using at the time, when [r] is followed by [a], I counted that there was a space inserted 631 times, and no space 573 times. That makes this glyph pair one of the most evenly split between spaced and unspaced instances. But I also counted 248 further cases shown as having ambiguous spacing (with a comma rather than a period / full stop). That's 17% of the total, which I believe also made this the glyph pair with the highest proportion of ambiguous spacing.

I considered two possible explanations:

• The ZL transcribers were more likely to judge a space as ambiguous when there was no consistent pattern corroborating whether it was "real" or not.
  
• The original scribe(s) were truly ambivalent about spacing between certain glyph pairs for some reason -- leading to inconsistency (some cases were clearly spaced, others clearly unspaced) or ambiguity (introducing a halfhearted sort-of-space).
  

René seemed pretty confident that explanation #1 wasn't the case, which would leave us with explanation #2.

I also found that these spacing irregularities weren't limited to specific "compound words" and "pairs of words" (so to speak), but seemed to affect all sequences containing the affected glyph pairs.

With [or] + [aiin], for example, I counted 26 instances with a space ["or aiin"], 29 instances without a space ["oraiin"], and 23 instances marked as ambiguous.  But substituting other words ending in [r] and/or starting with [a] typically produced a similar "mix" of spacings.

So this doesn't seem to be a situation where relationships between specific words are responsible for the glyph-level patterns.

The authors of this new paper write as follows in their introduction (the second citation is to You are not allowed to view links. Register or Login to view.):

 Transcribers have also long noted that the glyph pairs flanking uncertain spaces differ from those flanking ordinary spaces (You are not allowed to view links. Register or Login to view.; You are not allowed to view links. Register or Login to view.).

They later describe trying to test this in a couple different ways.

One is to compare the average gaps between voynichese.com bounding boxes for "certain" and "uncertain" spaces; the average size of an "uncertain" gap is smaller than for a "certain" gap, but we'd expect that even if the "uncertain" cases were simply a mix of real spaces and real non-spaces, so I don't know that we learn anything useful from that exercise.

But they also report finding that "certain" spaces tend to fall between their study's learned units, while "uncertain" spaces tend to fall within them. That seems potentially more meaningful in terms of supporting the idea of a hierarchy of "strong" and "weak" spaces.