Quant from Scratch by QuantWright

Backtesting Essentials · Lesson 7

The best of many

Why the best of many tries is probably luck, and what to do about it.

Know the basics? Read the lesson notes

Here is a trading strategy. It's a coin.

A trading strategy is just a rule that decides when to trade. This one is a coin: every flip is one trade. Heads, the trade wins 1R (+1R). Tails, it loses 1R (−1R). A fair coin lands heads half the time.

R is the amount you stand to lose on one trade. We count in R, so we never need a price. The account starts at 100R. Flip it ten times.

+1Rwin
−1Rloss

Flips 0  ·  account 100R

On average, what does one trade make?

Think about the next thousand flips, not the last ten. A strategy has an edge when it has a real reason to make money on average over time.

Only the winning half makes +1R. The losing half gives it all back: half +1, half −1, so the average is 0R.
Half the trades win 1R and half lose 1R, so the average is 0R.
A coin has no memory, so a streak doesn't change the next flip. Half +1, half −1: the average is 0R.
With an average of zero, this strategy has no edge. Whatever it makes is luck.

Now let it run for a year

Running a trading rule over a stretch of trades to see what it would have done is called a backtest. Here is your coin's backtest: the same coin, carried on from your ten flips to 250 trades, about a year if it traded once a day. Press play, then try a few other coins. Yours stays on the chart in a lighter line.

One coin-flip strategy, 250 trades

Account in R. Each step is one trade: up 1R on a win, down 1R on a loss.

Coin#1 (yours)
Trades0
Wins0
Account100R

Some finish well up, some well down. None of them can do anything but flip. Luck alone spreads results out, and over 250 trades it spreads them a lot.

A guess before we go big

We'll make 1,000 of these coin strategies and give each one 250 trades. Then we look at the best of them: the share of its trades it won, its win rate. A fair coin wins 50% on average. How high will the best one get?

Keep that guess in mind. Two screens from now you'll see the real answer.

How people tell luck from skill

Take one fair coin and flip it 250 times. Most of the time it wins somewhere near 125. Drag the marker to see how often a single coin wins at least that many.

One fair coin, 250 flips: how many wins?

The height of each bar is how likely that exact number of wins is.

A single coin gets there–

At 139 wins a coin gets there only about 4 times in 100. So when one strategy, tested once, wins 139 or more of 250 trades, it has done something luck alone rarely does, and people take that as evidence of an edge.

People draw the line where a coin gets there less than 5 times in 100, and call it the 5% line. It is a convention, not a law of nature. A result past it is called statistically significant.

Keep one thing in mind. The line is fair for one strategy, tested once. Next: 1,000 of them.

Now 1,000 coins

None of these coins has an edge. How many of the 1,000 will land on the 5% line or past it? Set your guess, then drop them in.

1,000 coin strategies, stacked by wins out of 250

Each block is one strategy. The dotted outline is what luck predicts.

Coins so far0
Fewest wins–
Most wins–
On the line or past it–

The line didn't fail. A single coin crosses it 4.4 times in 100 (that's the "less than 5" in its name), so about 44 of 1,000 coins should. We got 56, a little more: counts like this jump around from one drop to the next. The line was made for one test. We ran 1,000.

For the preciseExpected count on or past the line: 1,000 × 0.0438 = 43.8, with a standard deviation of √(1,000 × 0.0438 × 0.9562) ≈ 6.5; 56 is about 1.9 standard deviations above. The best coin: 155 wins, the worst: 100 (40%).

What a search does

Building a strategy usually means trying versions: a different setting (say, how many days of prices the rule looks back on), an extra condition before it trades, another market, hourly data instead of daily. Each version you try is one more coin in the pile. Then you keep the one with the best backtest. Slide from 1 version to 1,000 and watch the best one.

Try N versions, keep the best

Top: every version you tried (grey) and the best (colour), in R: a win adds 1R and a loss takes 1R from the 100R start, so 160R means 155 wins of 250. Bottom: the win rate the best of N coins reaches by luck alone, on average; each step along the bottom is ten times more versions. The more tries, the higher it goes.

Best version's win rate–
Best of N coins, on average–
Chance at least one crosses the line (each try: about 4 in 100)–

Nothing about the coins changed. Only how many you looked at. And "close to certain" has a reason: each try misses the line about 96 times in 100, and missing 1,000 times in a row almost never happens.

So what did you find?

Now leave our coins and picture your own search on a real idea. You tried about 1,000 versions of your rule (each one a different setting, an extra condition before trading, or another market, so each one is one more coin in the pile). Your best version won 60% of its 250 trades.

Past the line, yes, but the line was made for one test. The best of 1,000 fair coins wins about 60.2% on average: your result, with no edge needed.
Over 250 trades, the best of 1,000 fair coins wins about 60.2% on average, and any one search lands a little above or below that (ours: 62%). Your 60% is about what luck alone gives a search that size.
250 trades can tell you something. The trouble is the 1,000 versions behind the best: luck alone gives the best of 1,000 about 60.2%.

Pick one to trade

Here are the ten best backtests out of the 1,000. Choose the one you would trade. Then it gets 250 new trades it has never seen, flipped after the choosing was done. That's what happens when you start trading a winner, or test it on data you kept aside and never used to choose it (testing on data kept aside like this is called a held-out test).

The ten best backtests, then 250 new trades

Left: the backtest they were chosen on. Shaded, right: new trades.

Your pick, backtest–
Your pick, new trades?
All ten, new trades on average?

The ten averaged +44.2R in their backtests and −5.4R on the new trades. 4 of the 10 made money, about what any 10 coins would do. The best of many gets picked partly because of luck, and luck doesn't repeat.

All 1,000 at once

Each dot is one coin: how much it made in its backtest (across) and in its 250 new trades (up). Does a better backtest go with better new trades?

Look again: it doesn't tilt. If a good backtest told you something about the future, the cloud would tilt up to the right. For coins, the backtest says nothing useful about the next 250 trades.
If a good backtest told you something about the future, the cloud would tilt up to the right. It doesn't: a coin's backtest says nothing useful about its next 250 trades. That's what no edge looks like from the outside.
Theory

The chance that at least one no-edge try passes

A no-edge result that clears the line anyway is a false positive: the test flags it as more than luck, and it is luck. The 56 coins on the line or past it in your drop were 56 false positives. Running many tests and keeping whichever passes is called multiple testing (in trading research, data snooping).

If one no-edge try clears the line with chance p, it misses with chance 1 − p. Separate tries don't affect each other, so the chance that all n miss is 1 − p multiplied by itself n times, written (1 − p)n with a small raised n. The chance that at least one clears is 1 minus that.

The correction is to raise the line. Pick the most you accept being fooled across the whole search, α (the Greek letter alpha, usually 5 in 100), and give each of the n tests a line a no-edge try clears only α ÷ n of the time. Then n chances of α ÷ n add up to α, and the chance that at least one clears can't be more than that sum (two tries can both pass and get counted twice, so the real chance is a bit smaller: the sum is a safe upper limit). That is why the formula says ≈. This is the Bonferroni correction.

at least one clears: 1 − (1 − p)n
raised line, for each test: peach ≈ α ÷ n
p
the chance one no-edge try clears the line (0.0438 for 139 wins of 250)
n
the number of tries: every version you tested, kept or dropped
(1 − p)n
the chance all n miss
α
the most you accept being fooled across the whole search (usually 0.05)
peach
the chance a no-edge try clears the raised line
Worked example: your coins

The 139 line: p = 0.0438, so one coin misses with chance 1 − 0.0438 = 0.9562.

10 versions: 0.956210 = 0.639, so at least one crosses with chance 1 − 0.639 = 0.361, about 36 times in 100.

1,000 versions: 0.95621000 has 19 zeros after the decimal point, so at least one crossing is all but certain (the "over 99.9%" from "What a search does").

Raised line for 20 versions. It starts from the target α = 0.05, not from the 139 line's p = 0.0438: 0.05 ÷ 20 = 0.0025, or 0.25 in 100. That line is 148 wins of 250, and the best of 20 coins gets there about 4 times in 100. To check it yourself, press Back to "How people tell luck from skill" and drag the marker until "A single coin gets there" drops below 0.25% of the time: that happens at 148.

Holds when
  • no try has an edge (the formula counts lucky passes);
  • every try faces a line it clears with the same chance p;
  • the tries are separate: each coin ignores the others;
  • n counts every try, including the ones you dropped.
Breaks when

The tries are related: a 20-day and a 21-day setting trade almost alike, so they add less luck than two coins. Then the formula overstates the chance and α ÷ n is stricter than needed. Lesson B12 (The Sharpe ratio, and its error bar) counts how far a picked backtest's error bar falls short; a correction built for real searches comes in lesson C5 (Many tests at once).

A looser line, one a no-edge try clears 10 times in 100 (p = 0.1), and 5 no-edge versions. What's the chance at least one clears it? Type a percentage.

All five miss: 0.9 × 0.9 × 0.9 × 0.9 × 0.9 = 0.59. At least one clears: 1 − 0.59 = 0.41, about 41 times in 100.
One try misses with chance 1 − 0.1 = 0.9. All five miss: 0.9 × 0.9 × 0.9 × 0.9 × 0.9 = 0.59. At least one clears: 1 − 0.59 = 0.41, about 41 times in 100.
Raised for 5 tests, each would face a line a no-edge try clears 10 ÷ 5 = 2 times in 100.

What this means for your backtest

Three habits follow from everything above. Turn them over one at a time.

  1. Count every version you try.

    Every setting, extra condition and market you looked at is another coin in the pile, including the ones you dropped and forgot. Write them down as you go.

One last check

Why does a held-out test count only once?

Age isn't the problem. Once you've looked at the result and changed the rule, that period helped choose it: one more coin in the pile.
There's a real reason. Once you've looked and changed the rule, that period helped choose it, so it's one more coin in the pile.
Once you've looked and changed the rule, that period helped choose it, so it's one more coin in the pile.
Practice 1 of 4

The course seller's strategy

The situationA trading course shows a strategy that won 58% of its 250 backtest trades. In a footnote, the seller says they tested 200 versions before picking this one.

Before you pay, is 58% evidence of an edge?

Hint: press Back to the "What a search does" card and set its slider to 200 versions. The number under "Best of N coins, on average" is where the best of 200 coins lands.

The 5% line is for one test. With 200 versions, luck alone gives the best one about 58.7% on average over 250 trades: a little more than the 58% they show.
Over 250 trades, the best of 200 fair coins wins about 58.7% on average. A 58% best-of-200 is what luck delivers.
You can say a lot from the two numbers they gave you. The best of 200 fair coins wins about 58.7% on average over 250 trades, so 58% is no evidence of an edge.
The useful question to ask any seller: how many versions did you try, and how did the final one do on data it never saw?
Practice 2 of 4

Thirty settings, no edge

The situationYou test 30 settings of a rule that, unknown to you, has no edge: 30 coins, 250 trades each.

What's the chance that at least one of them lands on the 5% line or past it? Type a percentage.

Hint: each coin ignores the others. One coin misses the line 95.6 times in 100. All 30 missing is 0.956 × 0.956 × … thirty times. On a phone calculator: type 0.956, press the x^y key, type 30. About 0.26. Then: at least one crosses = 1 minus the chance that all 30 miss.

0.956 multiplied by itself 30 times is about 0.26, so all 30 miss about 26 times in 100, and at least one crosses about 74 times in 100.
0.956 multiplied by itself 30 times is about 0.26: all 30 miss about 26 times in 100. So at least one crosses about 74 times in 100. One coin is rarely lucky; one of thirty usually is.
Practice 3 of 4

Forty versions, one bar

The situationYou plan to test 40 versions of a rule and want the whole search to fool you no more than 5 times in 100.

Using the Bonferroni correction, each test's line should be one a no-edge try clears how many times in 100? Type a number.

5 in 100 ÷ 40 = 0.125 in 100 (0.00125, about 1 in 800). Check: 40 tests × 0.125 in 100 = 5 in 100.
Split the 5 in 100 across the 40 tests: 5 ÷ 40 = 0.125 in 100 (0.00125, about 1 in 800). Check: 40 tests × 0.125 in 100 = 5 in 100.
Practice 4 of 4

Your own research

The situationYou backtested 12 versions of an idea on 2018–2023 and kept 2024 untouched. Your best version looks good. You've written down all 12.

What do you do next?

2024 played no part in choosing, so one run there is a fair test. Whatever it says, it counts.
Picking the best of 12 on 2024 turns 2024 into one more search, and its best is partly luck again. Run only the version you chose, once.
More trades help, but folding 2024 into the search uses up your only untouched data. Then nothing is left to test the final pick on.
✓

Lesson done

Try many versions, keep the best, and part of its result is luck. Count the versions, raise the bar to match, and let a period you never touched have the last word.

Key ideas

5% line
a result a no-edge try reaches less than 5 times in 100; past it is called statistically significant. Fair for one test.
False positive
a no-edge result that clears the line anyway
Multiple testing
trying many versions and keeping the best (also: data snooping). Each try is one more chance of a false positive.
Bonferroni correction
to be fooled at most α of the time across the whole search (usually 0.05), raise the line so each of n tests clears it by luck only α ÷ n of the time
Held-out test
data kept aside while choosing, run once on the final version

Formulas

chance at least one of n no-edge tries clears = 1 − (1 − p)n

raised line, for each test: peach ≈ α ÷ n

Lesson notes

If you try many versions of a trading rule and keep the one with the best backtest (a replay of the rule over past trades), part of that result is luck, even when no version has an edge. The line a result must clear to count as significant (the 5% line, below) is set for one test, so a search of many versions needs a higher line. The last check is data you never used to choose.

A strategy that is a coin

The lesson's strategy is a fair coin: every flip is one trade. Heads wins 1R and tails loses 1R, where R is the amount you stand to lose on one trade. The account starts at 100R. Half the trades make +1 and half make −1, so the average trade makes 0R. A strategy has an edge when it has a real reason to make money on average over time. This coin has none, so whatever it makes is luck. A backtest runs a rule over a stretch of trades to see what it would have done. The lesson's coin runs 250 trades, about a year at one trade a day.

The 5% line is for one test

The win rate is the share of trades won. A fair coin wins 50% on average, which is 125 of 250. The chance that one fair coin wins 139 or more of 250 is 0.0438 (the lesson's computed chance, not worked by hand here), about 4 times in 100. People draw a line at the number of wins a coin reaches less than 5 times in 100 and call it the 5% line; here it is 139 wins. It is a convention. A result at or past it is called statistically significant: rare for a coin, though not proof of an edge. The line is fair for one strategy, tested once.

One thousand coins

In the lesson's simulation, 1,000 coin strategies each made 250 trades. None has an edge, yet 56 of them won 139 or more. A no-edge result that clears the line anyway is a false positive. The line did not fail: one coin crosses it 4.4 times in 100, so 1,000 × 0.0438 = 43.8 coins are expected to. The spread of that count is about 6.5 (√(1,000 × 0.0438 × 0.9562)), and 56 is (56 − 43.8) ÷ 6.5 ≈ 1.9 spreads above it. Counts like this jump around from one set of coins to the next. The line was made for one test, and the simulation ran 1,000. The best coin won 155 of 250 trades, a win rate of 62%.

What a search does

Building a strategy usually means trying versions: a different setting, an extra condition before it trades, another market. Each version is one more coin in the pile, and you keep the one with the best backtest. Trying many and keeping what passes is multiple testing (in trading research, data snooping). With 10 no-edge versions, the chance that at least one crosses the line is about 36 in 100. With 1,000 it is all but certain; the formula section below shows the working. In the lesson's simulation of fair coins over 250 trades, the average win rate of the best of n is 50% for 1 coin, 54.9% for 10, 57.9% for 100 and 60.2% for 1,000. So a best version that won 60% of its 250 trades, after about 1,000 versions, is about what luck alone gives the best of 1,000. Any one search lands a little above or below the average (the best was 62%).

New trades tell the truth

Take the ten best backtests out of the 1,000 coins. Their accounts rose by an average of +44.2R over the 250 backtest trades. Then each coin got 250 new trades, flipped after the choosing was done. On those, the ten averaged −5.4R, and 4 of the 10 made money, about what any 10 coins would do. Testing on data kept aside and never used to choose is a held-out test. Plot each of the 1,000 coins as a dot, its backtest result across and its result on the new trades up: the cloud shows no visible tilt. A coin's backtest says nothing useful about its next 250 trades, and any faint tilt is luck, since the new flips were drawn separately from the backtest flips. The best of many gets picked partly because of luck, and luck doesn't repeat.

Three habits

Count every version you try, including the ones you dropped. Raise the line to match the search. Keep some data you never touch and use it once: if you look at the result, tweak the rule and test again, that period has become one more coin in the pile.

Formula: at least one clears, and the raised line

If one no-edge version clears the line with chance p (in this lesson p is that chance, for example 0.0438 for 139 wins of 250, not a p-value), it misses with chance 1 − p. Separate versions don't affect each other, so the chance that all n miss is 1 − p multiplied by itself n times, written (1 − p)n. The chance that at least one clears is 1 minus that. The correction is to raise the line. Pick the most you accept being fooled across the whole search, α (alpha, usually 5 in 100), and give each of the n tests a line that a no-edge version clears only α ÷ n of the time. Then n chances of α ÷ n add up to α. The chance that at least one clears can't be more than that sum (two versions can both pass and get counted twice, so the real chance is a bit smaller: the sum is a safe upper limit). That is the lesson's reason for the ≈. This is the Bonferroni correction.

at least one clears: 1 − (1 − p)n
raised line, for each test: peach ≈ α ÷ n
p
the chance one no-edge version clears the line (0.0438 for 139 wins of 250)
n
the number of versions tried: every version you tested, kept or dropped
(1 − p)n
the chance all n miss
α
the most you accept being fooled across the whole search (usually 0.05)
peach
the chance a no-edge version clears the raised line
Worked example: the lesson's coins

The 139 line: p = 0.0438, so one coin misses with chance 1 − 0.0438 = 0.9562.

10 versions: 0.956210 = 0.639, so at least one crosses with chance 1 − 0.639 = 0.361, about 36 times in 100.

1,000 versions: 0.95621000 has 19 zeros after the decimal point, so at least one crossing is all but certain.

Raised line for 20 versions. It starts from the target α = 0.05, not from the 139 line's p = 0.0438: 0.05 ÷ 20 = 0.0025, or 0.25 in 100. The smallest number of wins that one coin reaches less than 0.25% of the time is 148 wins of 250, and the lesson computes that the best of 20 coins gets there 0.0424 of the time, about 4 times in 100.

For 1,000 versions: 0.05 ÷ 1,000 = 0.00005, or 0.005%, which a single coin reaches only from 157 wins of 250. The lesson computes that the best of 1,000 coins gets there 0.0306 of the time, about 3 times in 100, and the luckiest coin, at 155, no longer passes.

Holds whenBreaks when

The versions are related: a 20-day and a 21-day setting trade almost alike, so they add less luck than two coins. Then the formula overstates the chance and α ÷ n is stricter than needed. Lesson B12 (The Sharpe ratio, and its error bar) counts how far a picked backtest's error bar falls short; a correction built for real searches comes in lesson C5 (Many tests at once).

One more worked example

Made-up round numbers

You try 8 versions of a rule that has no edge. A line is set that one no-edge version clears 2 times in 100 (p = 0.02).

One version misses with chance 1 − 0.02 = 0.98.

All 8 miss: 0.988 = 0.851.

At least one clears: 1 − 0.851 = 0.149, about 15 times in 100.

To be fooled at most 5 in 100 across the search, the raised line for each test is 0.05 ÷ 8 = 0.00625, or 0.625 in 100. Check: 8 × 0.00625 = 0.05. The real chance that at least one clears is 1 − (1 − 0.00625)8 = 1 − 0.951 = 0.049, a little under 0.05, as the formula section said it would be.

Common mistakes

Check yourself

  1. One no-edge version clears a line with chance 0.2. You try 3 versions. What is the chance at least one clears?
    Answer

    One version misses with chance 1 − 0.2 = 0.8. All 3 miss: 0.8 × 0.8 × 0.8 = 0.512. At least one clears: 1 − 0.512 = 0.488, about 49 times in 100.

  2. You will test 25 versions and accept being fooled at most 5 times in 100 across the whole search. Using the Bonferroni correction, how often should a no-edge version clear each test's line?
    Answer

    0.05 ÷ 25 = 0.002, or 0.2 in 100. Check: 25 × 0.002 = 0.05.

  3. A seller shows a rule that won 58% of its 250 backtest trades, and says 100 versions were tried. Is that evidence of an edge?
    Answer

    It is not evidence of an edge. Over 250 trades the best of 100 fair coins wins 57.9% on average, and any one search lands a little above or below that. A 58% best of 100 is what luck alone delivers.

  4. Why does a held-out test count only once?
    Answer

    Once you have looked at the result and changed the rule, that period helped choose it, so it is one more coin in the pile.

Try it yourself

Run the simulation. Every line and number in this lesson comes from one Python script with a fixed random seed (below). Change the number of versions, the trades or the edge and see what moves.

Keep the count by hand. A spreadsheet is enough: one row per version you try, and the best judged against the average best of that many coins.

About QuantWright

QuantWright, which publishes this course, sells backtesting software, and two of its features are habits 1 and 3. Its Honesty Score (0–100) counts every distinct version your account has tested on that market in the last 30 days, so trying 50 variations does not make the best one look good. That count is a floor: versions you tried anywhere else aren't in it. It also runs a held-out test: it re-runs the chosen strategy once over years kept out of sight, and running it again in the same study no longer counts. In the app, a new chat is a new study with its own held-out test, so the discipline of habit 3 stays yours. quantwright.com

The code behind every number (Python, seed 2026110202)
Loading…

As a 71-second video: Backtesting from Scratch, Part 2, "The genius strategy that was pure luck". Same coins, same numbers.

Going further: a real edge, hidden among coins

Coins make the point cleanly, but you're looking for a strategy that does have an edge. So hide a real one among the coins. It wins 55% of its trades, which is +0.1R a trade on average (0.55 × 1 − 0.45 × 1): small, but real. To come out on top of a search it has to beat every coin, which means beating the best coin.

A real 55% strategy among fair coins

The coloured hill: where the real strategy's win rate could land. The grey hill: where the best coin's could land.

Trades
Fair coins in the search
Chance the real strategy finishes first–

With 250 trades and one coin beside it, the real strategy finishes first 86% of the time. Among 1,000 coins, only 5%: the other 95%, a lucky coin beats it and the search hands you the coin. More trades fix this. With 1,000 trades it wins the 1,000-coin search 46% of the time, and with 4,000 trades 99.8%, because both hills get narrower as trades add up and stop overlapping. Every extra version makes luck look better. Every extra trade makes luck look smaller. So prefer more trades: a rule that traded 30 times in ten years can't tell you much, however few versions you tried.

For the preciseP(real first) = Σk P(real = k) · F(k − 1)N, the real strategy binomial(T, 0.55), each coin binomial(T, ½), F the coin's distribution function; a tie counts as not first. T = 250, N = 1,000: 0.0509; T = 1,000: 0.4613; T = 4,000: 0.9979. The grey hill is the distribution of the maximum of N coins, F(k)N − F(k − 1)N, which is narrower than one coin's.
For the precise: the search and the scatterChance that at least one of N coins crosses: 1 − (1 − 0.0438)N; 36% at 10, 59% at 20, 99% at 100. Average best of N: E[best] = Σk≥1 (1 − F(k − 1)N). The correlation between backtest and new-trade results across the 1,000 coins is 0.10; for independent coins the true value is 0, and with 1,000 pairs the standard error is about 0.03, so 0.10 is rare luck (about once in 1,000 runs) but luck: the new flips were drawn after, and separately from, every backtest flip. Other names for this lesson's problem: the multiple testing problem, selection bias, data snooping.

Honest notes

Where this comes from