n-grams while ignoring spaces?

[obelus]

This discussion reminds me of an article in the 7 February issue of Science, "Whale song shows language-like statistical structure"  (Arnon et al., Science 387 (2025) 649).  It concludes that "subsequences" in humpback whale song follow a Zipfian power-law distribution.  A formal challenge along the way is to identify the boundaries between such subsequences.  Building on previous work in human infants, the authors segment whale-song recordings by identifying dips in transition probability between fundamental sound elements:  "Because words are statistically coherent, transitional probabilities within words are higher (on average) than those between words."  That the subsequences constitute a lexicon is "an assumption that cannot be reasonably applied to whales"—oh well—but their statistical structure is thought to facilitate learning and cultural transmission.

Relevant or not, the paper is entertaining, and the university has naughtily left a You are not allowed to view links. Register or Login to view. on their website.  For VMS researchers, it may suggest a new method of segmenting the text for analysis... or even calve a new theory of authorship.