The oddities of the bigram "ed"

[Jorge_Stolfi]

I typically strip all punctuation, lower case and then split the text into 100 word groups, something close to herbal page sizes.

A digram will stand out in your plots only if it occurs many times on a large fraction of the pages and zero times on another large fraction of the pages. Then the number of pages with the bigram will be anomalously low and it will stand out in the plot   

For example, suppose the bigram occurs 2000 times, and you have 100 pages, you would expect it to occur on almost all of those 100 pages.  But if instead there are 50 pages with four occurrences each, and 50 with no occurrences, the digram will stand out just like your ed

But if your "pages" are too small, even in the places where the digram is concentrated, almost all pages will have zero or one occurrences. Then the number of pages with the bigram will seem to be normal. 

Take that same text of the example above, but divide the text into 8000 pages instead of 100 pages.  Then those 50 pages that had all the 2000 occurrences will be split into 4000 pages, and thus those occurrences will be about one per page, and you will have ~2000 pages with occurrences.  Which is the expected number for 2000 evenly distributed occurrences.  Then the digram would seem to be normal in the plot.

Thus if a digram occurs N times overall, you should divide the text into, say, N/2 pages.  Then, if the digram is evenly distributed, one expects that it will occur in (1-1/e^2) = ~86% of the pages.   But if it is concentrated in some sections, the percentage of pages with the bigram will be much lower than that.

All the best, --stolfi


All the best, --stolfi