Your Mac version has a population comparison platform, which returns differences between my sample and control using four different algorithms. How can I compare them? Which one is better?
This question demonstrates a need for better documentation of this Platform. The short answer is - there is no way to cross-compare the results of these algorithms. I will discuss them briefly below..
The four methods for sample comparison are:
SED, Overton, K-S and T(x).
SED algorithm was developed and published by Bruce Bagwell, of Verity Software House. For more details on this please contact Dr. Bagwell.
Overton's subtraction algorithm goes through each bin of two histograms and subtracts the control from the exp. The resulting differences are summed and divided by total number of events - returning % difference. This algorithm does not provide any confidence metric for the value it returns.
K-S test is a very complicated math model developed to study the confidence in two distributions being significantly different. More on that here.
Finally, the T(X) method developed by Mario Roederer provides a calculated T value as well as the maximum possible T value. T(x) is a statistic which not only provides an indication of the probability with which two distributions are different, but simultaneously provides a metric by which multiple distributions can be ranked.
Users should be cautioned that K-S and T(x) will very readily label two samples as MATHEMATICALLY different. It's up to the user to determine whether this difference is BIOLOGICALLY meaningful.
T(X) test makes this a little easier by providing a maximum possible T value. T(X) = 4 is the same significance whether the maximum value is 50 or 50,000 -- it implies that you are 4 std deviations away from expected. The reason the max T(X) can range is that with more events (or more bins) you can achieve greater precision -- thus, greater significance. For example, if you do a wilcoxon rank test on two groups of 3 events, the most significant p value you can achieve is about 0.03 -- when there is no overlap between the two groups. With 4 events, the most significant p value you can achieve is about 0.004. But a p value of 0.05 is the same significance whether the best significance you can achieve 0.03, 0.004, or 10^-44. Similarly with T(X) -- a T(X) = 4 is the same; the max(T(X)) just tells you what the maximum significance could have been.
In sum, don't use software to find differences in your flow data. Use it to substantiate and quantitate the differences you already know about.
Comments