Distribution of Q-Q gaps in paragraphs
#15
(22-01-2026, 09:25 PM)dashstofsk wrote: You are not allowed to view links. Register or Login to view.Again, under a meaningful text hypothesis distributions mid-line should be level

You mean under a random gibberish hypothesis.  

In meaningful text one expects uneven distribution of words along sentences.  Depending on the content of the parags, one expects the distribution of words along parags to be uneven too.   Like, in a herbal, the word "herb" being more common at the beginning, while "grows" more common close to the end

There are various hypotheses that could explain how an uneven distribution of words in sentences and parags can produce uneven distribution along a line.

Some manuscripts have peculiar devices to help the reader or other reasons.  Like placing a ¶ at the beginning of a line that contains the start of a new sentence or topic.  Others may put such a mark inside the line, unless the sentence starts at the beginning of the line...

And please folks, this is important: when text is formatted into paragraphs, longer words will be be more common at start of line, and less common at the end of line.  In any language, any text, any spelling or encoding.  That will affect the distribution of prefixes, glyphs, digraphs etc at those places.

Indeed 320 parags are too few to estimate the frequency of a word or prefix at the start or end of parags.  But if you take 320 words at random  out of the VMS lexicon, the probability of them being all distinct is very low.  Several words occur with frequency 1% or more. You would expect quite a few duplicates...

All the best, --stolfi
Reply


Messages In This Thread
RE: Distribution of Q-Q gaps in paragraphs - by Jorge_Stolfi - 23-01-2026, 02:51 AM

Forum Jump:


Users browsing this thread: 1 Guest(s)