How to claim A > B

**lzhang** Tue Jul 03, 2007 2:57 pm

How to claim A > B
If there are two sets of series, A(a1, a2, a3, ..., an) and B(b1, b2, b3, ..., bn), the mean of A is a, and the mean of B is b, how could we claim E(A)>E(B). Can we just say the mean of A is definitely greater than the mean of B just because a>b? Obviously not.

Actually it is a simple hypothesis test in statistical language.

null hypothesis H0: μ(A)≤ μ(B)

alternative hypothesis Ha: μ(A)>μ(B)

If A and B are series of investment returns, usually a's and b's are correlated, so it is better to combine them together to have C(a1–b1, a2–b2, a3–b3, ..., an–bn), and now the test becomes

null hypothesis H0: μ(C)≤ 0

alternative hypothesis Ha: μ(C)>0

The test statistic follows t-distribution. If we require significance level at 0.05. For df=4–1=3, we need t>2.4 to reject the null hypothesis. For df=12–1=11, we need t>1.8 to reject the null hypothesis. For df=42–1=41, we need t>1.7 to reject the null hypothesis.

Buffett Partnership has 12 sample points, the t value for the test is t=4.14, which is far greater than 1.8, to put it in another way, Buffett Partnership's return is very very significantly greater than DOW return at a level far exceeding our test criterion (we can say that there is very very strong evidence on the claim).

BRK has 42 sample points, the t value for the test is t=4.8, which is far greater than 1.7, to put it in another way, BRK's return is very very significantly greater than SP500 return at a level far exceeding our test criterion (we can say that there is very very strong evidence on the claim).

Please remember t=4.8 for such a sample is a very remarkable number. In particle physics, we usually can claim that we discover a new particle or a new channel if we have t ≥ 5. So the claim that BRK has higher performance than SP500 is almost as solid as a claim on that we have proton.

With only 4 sample points, the t value of OAKLX for the test is t=1.99, which is less than 2.4, which means we can not reject the null hypothesis, i.e., OAKLX has less or equal return as SP500, based on the 4 sample points we have, on a significance level of 0.05 (we can say that there is no strong evidence on the claim).

The same thing for portfolio X with t=1.75, which means we can not reject the null hypothesis, i.e., X has less or equal return as SP500, based on the 4 sample points we have, on a significance level of 0.05 (we can say that there is no strong evidence on the claim).

However, if we lower the significance level requirement to 0.10, i.e., and ask `do we have some evidence on the claim?' For df=3, if t>1.6, we can reject the null hypothesis. So we can say that based on the 4 sample points, we have some evidence on the claim for both OAKLX and X.

The problem here for the last 2 portfolios is the sample size is too small, which means we can not make much statistically meaningful conclusion. On the other hand, we can trace OAKLX back with more data, so we can have more decisive conclusion, but for portfolio X, we only have returns in 4 years. Unless the differences between X and SP500 are large and consistent, we can not have strong claim on the performance based on current past data.

http://spreadsheets.google.com/pub?key=pBWqCNQoWBZo2bOHCtBIGcQ

http://spreadsheets.google.com/pub?key=pBWqCNQoWBZrbt0K3YEQcbw