Chronological Path Stability/Uncertainty Analysis

[npcompl33t]

Here "row" means a left character g, correct? So you consider only digraphs of characters g that occur at least three times on the page? 

Yes, correct, or more precisely 'starting glyph unit g'. The threshold is 3 outgoing transitions from g, not three occurrences of each bigram dh.

Let G(p) be the set of g characters that qualify for a page p.  Then for any page p and any g in G(p) you have a distribution H(p,g), a vector of M probabilities P(h|g), where M is the size of the alphabet you are using.  Check?  

Yes.

How do you combine those vectors into the vectors of 256 numbers that you feed to MDS?

MDS is not fed a 256d feature vector per page. Classic MDS is done by pairwise dissimilarity of the page vectors, producing an NxN matrix, that MDS uses to produce the coordinates.
  

Any way you estimate the probabilities P(h|g), the result will have an uncertainty that depends on how many times g occurs in the page.  If on two pages p,q there are 10 occurrences of "g" but 3 and 4 of "gh", respectively, intuitively those two pages are more similar on that particular digraph than if they had 1000 "g" and 300 and 400 "gh".  But the JS divergences and distances use the probabilities P(h|g) which are 0.30 and 0.40 in both cases.  


This obliviousness to the counts is already a problem for interpretations.  But the sensitivity of JS metric to small probability values makes the problem worse.  That might justify using the Bayesian estimates of P(h|g) instead of the simple frequencies, since it would ensure that all probabilities are nonzero.

In particular, if g does not occur on a page at all, the Bayesian P(h|g) will be 1/M for all h (not 0/0).  Then you should not have to exclude any rows. But ideally one should also keep track of the uncertainties, and use them to compute weights for MDS, curve fitting, etc.  But that looks like a huge complication... 

 

Actually, thanks, this is a great suggestion. 

It is substantially more complex, but I think it might be a genuinely more accurate picture. What do you think of something like this:

- Use each pages transition counts to estimate a posterior distribution for each row
- Draw alternative probability profiles from those posteriors
- Recompute the dissimilarities and MDS for each row
- Refit curves 
- Report whether the horseshoe survives uncertainty in the underlying profiles. 

If I were to do the above, would that be enough to settle the matter for you? 

If the sections are mapped to points of R^256 and measured with the Euclidean metric,  alignment would mean the obvious thing -- they lay near some straight line of R^256.  But we could also compare their prob vectors with JS distance and check how much Jdist(A,B) + Jdist(B,C) is equal to Jdist(A,C).  This alignment criterion could disagree with the Euclidean one.  Which one would be right?

Jdist(A,B) + Jdist(B,C) means the embedded B lies on the straight segment between embedded A and C. 

Can JS distance be represented as Euclidean DIstance? Yes, in Hlibert space, but not generally in R^D or R^D-1.

If this mapping existed, then we could apply it and work with Euclidean metric -- which we understand better.  Or less poorly...
[/quote]

Classical MDS using JS divergence can construct an exact Euclidean representation. I could probably figure out a way to adjust my exact distance formula to one that would work. Maybe something like a fixed weighted sum would work i think. I would be happy to add it into the above Bayesian model if you feel that it would help convince you of the results. But if I do this and it comes back as a horseshoe or gradient will you accept that the gradient exists?


Your own 2000 paper seems to suggest that it does. Didn't you find that the word length distribution is almost perfectly binomial? How would the entire text have a binomial length distribution if sections are driven by different topics with different vocabularies? Every section individually must have a binomial distribution, what changes is the word families, not the construction. This is consistent with a gradient over the course of the entire MS, not separate topic driven vocabularies. 

Sorry, which one was that?  You did several...

You are not allowed to view links. Register or Login to view. Nablator even answered your question and gave you an exact list of mappings.