Backtesting Essentials · Lesson 6
Hypothesis tests
Luck or edge? Ask how often pure luck would do as well as your strategy did.
Know the basics? Read the lesson notesIs this a coin?
A trading strategy is a rule that decides when to trade. Here are six made-up strategies, each with its record: how many of its trades won, out of how many it made. The share it won is its win rate. In all six, a win and a loss are the same size, so only the number of wins matters.
A strategy with no edge (no real reason to win more often than it loses) is like a fair coin. It wins half its trades on average, but any one record lands a bit above or below half. For each record: could a coin have done this, or would it be hard for a coin?
Keep your answers. Near the end of the lesson you'll check all six.
Start by assuming it's a coin
Take strategy C: 136 wins of 250 trades, a win rate of 54.4%. A coin wins 50% on average, so 54.4% looks better. But a coin's record also wanders above and below half.
So start from the dull explanation: suppose C has no edge and is just a coin. A coin is a case where we can work out exactly what luck does; "it has some edge" doesn't say how much, so it gives no single number. We test luck first. Then ask how often a coin would do this well. Find out by running a coin: 250 trades, each a 50–50 win or loss. Run it five times.
A no-edge coin, 250 trades a run (simulated)
Each block is one run, placed at the number of trades it won; the black block is this run. The line marks strategy C's 136.
A guess before 10,000 runs
Next, the same coin runs 10,000 times, 250 trades each. How many of the 10,000 runs will win 136 trades or more, as many as strategy C did?
Run it 10,000 times
Each run is one no-edge strategy that could have happened. Run all 10,000 and watch where they pile up. Runs that won 136 or more are coloured.
10,000 runs of a no-edge coin, 250 trades each (simulated)
The height of each bar is how many runs won that many trades.
That share has a name: the p-value. It answers one question: if the strategy had no edge, how often would luck alone do at least this well?
About 1 time in 11 is not rare. A coin often does what strategy C did, so C's record is weak evidence of an edge at best. It isn't proof that C has no edge either: this record simply can't tell the two apart.
Where does rare begin?
People call a result rare when fewer than 5 runs in 100 reach it, a convention. Drag the marker along the same 10,000 runs: the further right you go, the fewer runs reach it. Find the first number of wins that fewer than 5 runs in 100 reach.
The same 10,000 no-edge runs: how many reach the marker?
Coloured: the runs that won at least as many trades as the marker. Drag on the chart or use the slider.
At 139 wins (a win rate of 55.6%), 467 of the 10,000 runs get there: 4.67%, under 5 in 100. At 138, 605 runs still do: 6.05%.
A result with a p-value under 5% is called statistically significant: luck alone does that well less than 5 times in 100. This is the 5% line. 5 in 100 is 1 in 20.
Keep one thing in mind. The line is fair for one strategy, tested once. What happens when you try many is the next lesson.
For the precise
Counting every possible sequence of 250 coin flips exactly, instead of running 10,000 of them, a coin wins 139 or more with chance 0.0438 (4.38%) and 138 or more with 0.0568. The 10,000 runs give 4.67% and 6.05%: a share from 10,000 runs wobbles by about 0.2 percentage points from one set of runs to the next. Both put the line at 139.What a p-value doesn't say
Picture one strategy, tested once. Its test gives a p-value of 3%. What does that tell you?
Back to your six records
Now the same test on all six. For each record, the coin runs 10,000 times with that record's number of trades. Pick a record to see where its luck lands and its p-value. Look at all six.
10,000 no-edge runs for each record's number of trades (simulated)
The axis now shows win rate (wins ÷ trades), so different trade counts line up. Bars: how many runs landed there. Coloured: the runs at least as good as the record.
A and D both won 70%, and a coin still matches them: A about 17 times in 100, D about 5.4 times in 100, so D just misses the line. Over 10 or 20 trades luck has lots of room. B won only 54%, yet over 1,000 trades a coin rarely strays that far from half: about 0.7 times in 100. The fewer the trades, the wider luck spreads, and the further a win rate has to be from 50% before it is rare for a coin.
The null hypothesis and the p-value
Every test on this page followed one recipe, called a hypothesis test. Start from the null hypothesis: the dull explanation, here "no edge, just a coin". See what results it produces, then where yours falls. The p-value is the share of those results at least as good as yours. If it is under a line set before you look, the significance level α (alpha, usually 5%), the result is statistically significant. The line comes first because a line picked after seeing the result can be moved to suit it.
- no-edge runs
- results the null hypothesis produces: here, a coin with the same number of trades
- α
- the significance level, chosen before the test: usually 0.05, the 5% line
This counts one side only, a one-sided test, because the question is "does it beat a coin?". A two-sided test asks "is it different from a coin, better or worse?" and also counts runs as far from half on the losing side. Because the bars are the same on both sides, that about doubles the p-value. A p-value is not the chance the strategy works, nor the chance the result was luck: it is worked out by assuming luck.
945 of the 10,000 runs won 136 or more: p = 945 ÷ 10,000 = 0.0945, or 9.45%. That is above 5%: not significant.
Two-sided: 136 is 11 wins above the coin's 125, so also count runs 11 or more below it, at 114 wins or fewer. 1,874 runs land on one side or the other: 18.74%, about double.
- the no-edge runs match the real test: the same number of trades, the same win and loss sizes;
- one strategy is tested once, and the test and the line are chosen before you look;
- trades are independent: one trade's result doesn't change the next one's.
You try many versions and keep the best. Each try is another chance for luck to pass, so the 5% line stops meaning 5% (lesson B7, The best of many).
A different test: someone ran 2,000 no-edge versions of it, and 34 did at least as well as the real result. What is the p-value? Type a percentage.
A shortcut: the z-test for a win rate
For a win rate there is a shortcut. A coin's win rates pile into the bell curve (lesson A5), centred on 0.5. Its width is the standard error from the last lesson, √(0.25 ÷ n) for a coin. Measure how many standard errors your win rate sits above 0.5 (its z-score), then read the chance off the bell's tail, its thin end past that z-score: that is the z-test.
For the precise: wins and win rate
The bars on "Run it 10,000 times" count wins; the formula measures the win rate, wins ÷ n. For 250 trades the pile of bars has a spread (one standard deviation) of about 7.9 wins (√250 × 0.5 = 7.91; the 10,000 runs measured 7.96), and 7.91 ÷ 250 = 0.0316: the same number the formula gives, √(0.25 ÷ 250) = 0.0316.- p̂
- "p-hat": your win rate as a decimal (54.4% is 0.544)
- 0.5
- a coin's average win rate
- 0.25
- 0.5 × 0.5: a coin's chance to win times its chance to lose (the p × (1 − p) in the previous lesson's standard error, with p = 0.5)
- n
- the number of trades
- z
- how many standard errors the win rate sits above 0.5
On one side, 5 in 100 of the bell lies past z = 1.645, so that is the one-sided 5% line. A two-sided test splits its 5 in 100 into 2.5 on each side, and 2.5 in 100 lies past 1.96, so its line is at 1.96. Drag to read the tail for any z:
The bell's tail past z
Coloured: the share of the bell past z, on one side.
60 wins of 100: p̂ = 0.60. Standard error: √(0.25 ÷ 100) = √0.0025 = 0.05. z = (0.60 − 0.50) ÷ 0.05 = 2. Past 1.645: significant. The bell says about 2.3 in 100; the runs counted 274 of 10,000, or 2.74 in 100. Close: the bell is a smooth stand-in for the bars.
Strategy C, 136 wins of 250: p̂ = 0.544, standard error 0.0316, z = (0.544 − 0.500) ÷ 0.0316 = 1.39. Short of 1.645: not significant. The bell says about 8.2 in 100; the runs counted 9.45: the bell reads a little low here because the bars are whole steps and the smooth bell misses part of the step it starts on (more below).
- every trade wins or loses the same amount, so only the win rate matters;
- there are enough trades: a few dozen or more, and more still when the result sits close to the line;
- one strategy is tested once.
Trades are few. For strategy D, 14 of 20, z = 1.79 is past 1.645, yet the runs counted 542 of 10,000, or 5.42 in 100: the wrong side of the line, because with few trades the bars are chunky and the smooth bell misses them. Trades of different sizes need a test of the average trade instead (lesson C4, Bootstrap and permutation tests).
A rule won 1,325 of 2,500 trades (53%), every win and loss the same size. What is its z-score?
What this means for your backtest
A backtest runs a trading rule over past prices to see what it would have done. Three habits follow. Turn them over one at a time.
- Start from "it's a coin".
Before you believe a backtest's win rate, ask how often a no-edge strategy with the same number of trades would do at least as well. That share is the p-value. The coin fits rules whose wins and losses are the same size; a rule that wins 3 for every 1 it loses needs a different no-edge comparison.
- Significant means rare for luck, not proven.
Under 5 in 100 is a convention: 1 coin in 20 passes it by luck. And a p-value never gives the chance that the rule works.
- One test, set up before you look.
The 5% line is fair for one strategy, tested once, with the test chosen in advance. Try many versions and keep the best, and luck gets many chances to pass.
A friend's luck machine
What is the p-value, and is the result significant?
Your own test
What is its z-score? Type a number.
Hint: the standard error is √(0.25 ÷ 400). Then divide (0.56 − 0.50) by it.
Two sellers
Whose record would be harder for a coin to match?
Lesson done
To judge a result, start by assuming it is luck, and ask how often luck alone would do at least this well. That share is the p-value. Under 5 in 100 is called statistically significant: rare for luck, not proof of an edge, and fair only for one test set up before you looked.
Key ideas
- Edge
- a real reason to win more often than a coin; with no edge (and wins and losses of equal size), a strategy wins half its trades on average
- Null hypothesis
- the dull explanation a test starts from: no edge, just a coin
- Hypothesis test
- see what results the null hypothesis produces, then where yours falls
- p-value
- the share of no-edge results at least as good as yours. Not the chance the strategy works.
- Statistically significant
- a p-value under the significance level α, usually 5% (the 5% line). Fair for one test.
- One-sided, two-sided
- "does it beat a coin?" counts one side; "is it different from a coin?" counts both
- Standard error
- how far luck typically moves a measured win rate: √(0.25 ÷ n) for a coin
- z-test
- a win rate's distance from 0.5 in standard errors, read off the bell
Formulas
p-value = (no-edge runs at least as good as yours) ÷ (all no-edge runs)
z = (p̂ − 0.5) ÷ √(0.25 ÷ n)
The 5% line: p-value under 0.05, or z past 1.645 one-sided (1.96 two-sided).