Proposed Manuscript Chronology

[Jorge_Stolfi]

 For a horesshoe to mechanically appear, a single linear ordering, must be the most significant part of the feature space. Otherwise it would not  appear. By appearing, it means there is a meaningful order.

There seems to be a lot of superstition and quackery in the field of data analysis.  

For example, there is a big sub-area (called Technical Analysis or TA) devoted to analysis of stock prices, where the main goal is to predict the price P[t+1] of a stock at time t+1 (next month, day, millisecond) from its past history P[t], P[t-1], ....  TA practitioners have defined dozens of patterns that are supposedly important for that goal, such as "trends", "flags", "support", "MACD", "head and shoulders", "EMA crossings", "Fibonacci retracings", "Anchored VWap", etc.  And there are thousands of thick books about TA, both general and devoted to pushing a particular tool.  

But 99.99% of that theory is nonsense -- produced by the same motives that drive quacks in any field, and sustained by pareidolia, wishful thinking, and that old human desire:

• Tiger got to hunt, bird got to fly;
  
• Man got to sit and wonder 'why, why, why?'
  
• Tiger got to sleep, bird got to land;
  
• Man got to tell himself he understand. (K. Vonnegut)
  

The 0.01% of TA that is not nonsense is the single result that is supported by proper analysis, the Geometric Brownian model.  Which says that log(P[t+1]) = log(P[t]) + R[t], where each R[t] is a random variable with approximately Gaussian distribution, independent of all previous prices P[t-1],P[t-2], ...  That is, stock prices are effectively unpredictable, and their past histories do not give any information about their future, except that the price tomorrow will be somewhat similar to the price today.

Well, that "Guttman Horseshoe" sounds like "superstition" too.   Consider this fact: Let c[1], c[2],... c[m] be any m data points of R^d, and p[1], p[2], ... p[m] be any m points of the plane R^2, in any order.  If d >= m, there is an affine  projection of R^d to R^2 that projects each c[k] to p[k].

(An affine projection is a linear projection plus a translation.  The claim is false in some degenerate cases, when the data points happen to lie all in a small subspace of R^d; but this should not be the case for the VMS page data.)

In particular, for any 5 clouds of points in R^d with d >= 5, and any permutation of the indices 1..5, there is an affine projection of R^d to R^2 that will map the centroids of those clouds to five points on the parabola y = x^2, in that specified order and equally spaced in x.  In other words, we can always get a "Guttman Horsehoe"  with the clouds in any arbitrary order.

Here is an example with d = m = 4.  The data points are 4 clouds whose centroids are the vertices of a regular tetrahedron in R^4.  Within each cloud the points are drawn from an almost round 4D Gaussian distributon. So there is no intrinsic order in the data; all 4! = 6 orders of the 4 clouds are equivalent.  

Here is a projection of those data points where the four centroids project to the corners of a square:
   

But here is another projection where the same clouds lie on a "horseshoe" curve, apparently in order red-green-cyan-purple:
   

All the best, --stolfi