Chronological Path Stability/Uncertainty Analysis

[Jorge_Stolfi]

Your own 2000 paper ... Didn't you find that the word length distribution is almost perfectly binomial? How would the entire text have a binomial length distribution if sections are driven by different topics with different vocabularies?

The binomial-shaped distribution of lengths ("binomial distribution" was a misnomer, sorry -- it means something else) is about the lexicon.  That shape appears in the East Asian languages and in Roman numerals because of their word structure: several slots, each of them having a small set of possibilities, including empty -- and a large fraction of all possible combinations actually occurring in the text.

If one instead plots the length distribution of tokens in Vietnamese (or Voynichese), the curve comes out lopsided -- because in Vietnamese, as in any natural language, the most common words tend to be shorter than the less common ones.

I don't recall computing the word length distribution separately for each VMS section.  But I expect that it would be binomial-like too.  Because the lexicon of each section will be a large subset of the whole lexicon, and whether a word of the global lexicon is used or not in a section would hardly depend on the word's length.  

The length would of course influence how often that word will be used in the section; but this has no effect on the word length plot, because it will count that word only once.

Whereas the digraph frequencies are computed from occurrence counts, and therefore are dominated by the presence or absence of each digraph in the most common words.  If two sections X, Y differ mainly in the fact that "aardvark" is very common in in X but rare or absent in Y, that would have a significant impact on the "a" row of the transition count matrix. 

Classic MDS is done by pairwise dissimilarity of the page vectors, producing an NxN matrix, that MDS uses to produce the coordinates.

Ah, OK.   Do you give it the JS divergences, or the JS distances?

What do you think of something like this:
- Use each pages transition counts to estimate a posterior distribution for each row
- Draw alternative probability profiles from those posteriors
- Recompute the dissimilarities and MDS for each row

Sounds better (and eventual referees of your paper may like it more).  

- Refit curves
- Report whether the horseshoe survives uncertainty in the underlying profiles. 

I don't care about the curves or the horseshoe.  I tried to explain before why I think that they are meaningless. And anyway they will be a poor way to analyze the data.  Like trying to understand a theater play from the shadows that filter through the gap under a closed door.  

Again, you can always fit a curve to any set of data points is any space R^n.  Projecting the points onto the curve will define an ordering for them.  But that curve and that ordering will not mean anything.

IF it turns out that the points are all close to a straight line (or even to a parabola, or some other simple curve), then that would be a notable fact.  It would mean that there is a single parameter t that largely determines the word frequency distribution of the pages.  However, the points would have to be close to that line or curve in R^n space, not just in some projection -- even if that projection was chosen by some "agnostic" method like MDS.

But I have not seen any sign of such alignment.  I still see only that the page points are clustered into 4-5 major clouds, each apparently Gaussian-like, whose centroids seem to be in "random" positions of R^n space.

In fact, I am not very interested in the "main" directions chosen by MDS, because they can be radically changed by re-encoding of the data, even by a simple linear transformation of R^n -- and I still don't understand enough to tell whether the current encoding is the correct one, or whether the coordinates (probabilities) should be modified, e.g by weights or by combining characters into classes. 


I am interested in the clustering of the pages, and the separability of the clusters; and for that the 2D projections, MDS or not, are better (or "less worse") than the 1D projections onto a snaking curve.   Note that clustering and linear separability are invariant under arbitrary linear transformations of the data.

If I were to do the above, would that be enough to settle the matter for you?

I admire your optimistic view of human nature. 

Jdist(A,B) + Jdist(B,C) means the embedded B lies on the straight segment between embedded A and C.

It means that B lies between A and C along the JS geodesic that goes through these two.  But didn't you say that the JS geodesics are not straight?

(If the geodesics were straight lines in R^n, I think one could prove that the JS distance is just a multiple of the Euclidean distance.  Which is not the case, right?)

Classical MDS using JS divergence can construct an exact Euclidean representation. I could probably figure out a way to adjust my exact distance formula to one that would work. Maybe something like a fixed weighted sum would work i think.

I suspect that it would have to be a non-linear mapping from the canonical probabilities simplex to some higher-dimensional space.  Because (IIUC) Jdist becomes more senstive near the boundary of the simplex.  That is, the mapping F would have to stretch the parts of the simplex near the boundary, maybe off to infinity.

It would be useful to understand this question for probability vectors on n=2 or n=3 items only.  For n=2, the simplex is just the segment from (1,0) to (0,1).  What does Jdist((p,1-p), (q,1-q)) look like, as a function of p and q? How does it compare to the Euclidean distance Edist((p,1-p), (q,1-q)) = |p-q|*sqrt(2)?. 


All the best, --stolfi