Quant from Scratch by QuantWright

Backtesting Essentials · Lesson 13

Benchmarks

A result means little without a benchmark to measure it against. For a rule on one market, the natural one is just holding that market over the same days, with the same costs.

Know the basics? Read the lesson notes

It made 59%

A backtest runs a trading rule over past prices to see what it would have made. Here the prices come from a simulated market. Rule 1 holds the market when its close is above the average of its last 200 closes, and holds cash otherwise. It decides at each day's close (its last price) and trades at the next day's open (its first price). Every buy and every sell pays 0.1% of the amount traded in costs (lesson A16).

From the start of 2016 to the end of 2024, nine years, rule 1 turned $10,000 into $15,912: +59% after costs. Is that good? Guess what just holding made: buy the market at the first open of 2016, never sell, and pay the 0.1% cost once, for the one buy.

Rule 1 on a simulated market, 2016–2024

Colour: rule 1, after costs. Dark: just holding, after its one buy (shown once you guess). Dashed: the $10,000 start.

Holding made:
Rule 1+59%
Just holding?

Just holding turned $10,000 into $28,353: +184%, far ahead of rule 1's +59%. On its own, +59% said nothing. Just holding is the benchmark here: a simple alternative that a rule is measured against.

Four rules, one market

Three more rules ran on the same market over the same days, with the same costs. Rule 2 holds when the close is above the average of its last 20 closes. Rule 3 holds on exactly the days rule 2 doesn't. Rule 4 holds every day except the day after a close 2% or more below the close before. For each rule, decide: did it beat just holding? Then turn holding on.

Four rules on one simulated market, 2016–2024

Each bar: the rule's result after costs. Dashed, once turned on: just holding, after its one buy.

Rule 1, +59%:
Rule 2, +53%:
Rule 3, +7%:
Rule 4, +159%:
Holding:

All four made money, from +7% to +159%, and none beat holding's +184%. Their leads over holding, each rule's result minus holding's in percentage points (points, for short: the plain difference of two percents, as in 59 − 184 = −125), were −125, −131, −177 and −25 points. Compared with zero, all four look like successes. Compared with holding, none is.

Theory

The lead over holding

For a rule on one market, the usual benchmark is just holding that market: the rule that is in on every day. The lead is the rule's result minus holding's, in percentage points.

lead = net − hold
net
the rule's result after costs over the test days, in % = (end value ÷ start value − 1) × 100
hold
just holding's result on the same market over the same days, after its one buy, in % = (end value ÷ start value − 1) × 100
lead
percentage points ahead of holding; a minus sign means behind
Worked example: rule 4

Rule 4 made +159% after costs, and holding +184% after its one buy. lead = 159 − 184 = −25: 25 points behind holding, although it made 159%.

Holds when
  • both run on the same market over the same days;
  • both pay the same costs: 0.1% on every buy and sell, which for holding is its one buy;
  • both reinvest the same way: here, every account puts its whole value in at each buy.
Breaks when

Any of the three differs (the card "Like with like"). And a lead from one market is one draw of luck, not the rule's lead on every market; the next evidence is a held-out test (running the rule once on days it was never tuned on: the Held-out test lesson).

New numbers. A rule's account went from $10,000 to $13,400 after costs. Just holding the same market over the same days went from $10,000 to $15,100 after its one buy. What is the rule's lead over holding?

net = 3,400 ÷ 10,000 = +34%. hold = 5,100 ÷ 10,000 = +51%. lead = 34 − 51 = −17: 17 points behind.
First each result in percent: the rule gained 3,400 on 10,000, +34%; holding gained 5,100, +51%. Then lead = 34 − 51 = −17. The minus sign matters: the rule is 17 points behind.

Made money, or beat holding?

One market is one draw of luck. Here are 1,000 more simulated markets over 2016–2024, each drawn by the same generator: the computer recipe that drew the lesson's market. Each rises slowly on average. Past prices say nothing about which way the next move goes; only the size of moves clusters, into calm and wild spells (runs of small moves and runs of big ones). So no rule here has anything real to find. Rule 1 made money (ended above $10,000 after costs) in 752 of them. Guess: in how many did it beat holding?

Four rules on 1,000 simulated markets

Grey: markets where the rule made money. Colour: markets where it beat just holding. Both after costs.

Your guess:

Rule 1 beat holding in 138 of the 1,000. Every count here is after costs. Rule 2 made money in 615 and beat holding in 59; rule 3, 541 and 40; rule 4, 853 and 155. Holding itself made money in 900.

Why so few? On a day a rule holds cash it misses that day's move. These markets rise on average and a rule here can't pick its days, so the days it misses take part of the rise away from it. Even before costs, rule 1 beat holding in only 174 markets. The less a rule was in the market, the further it trailed: rule 4, in on 97% of days on average, had an average lead of −37 points after costs, and the three rules that were in less often trailed further.

For the preciseAveraged over the 1,000 markets, after costs: rule 1 was in on 63% of days with an average lead of −70 points; rule 2, 55% and −110; rule 3, 45% and −123. Rules 2 and 3 also bought or sold about 31 times a year, so costs took more from them.

Rules 2 and 3 show the first reason plainly. On every day exactly one of them holds the market. On the lesson's market, before costs, rule 2 turned each $1 into $2.011 and rule 3 into $1.411, and 2.011 × 1.411 = 2.838: what holding turned $1 into. They split holding's rise between them.

Like with like

A lead counts only if the two results were measured the same way. On the lesson's market, rule 4 trailed holding by 25 points. Here are three common mismatches. Look at each.

Rule 4 against just holding, one simulated market

Both after costs, same days, both reinvesting.

Other dates. Holding measured from the first open of 2015, when the data starts, a year before the backtest's first day (the rule needs a year of past closes before it can start): +95%. On this market 2015 was a falling year, which that holding has to carry and the rule never did. The lead becomes 159 − 95 = +64.

Costs left out. Rule 4 before costs made +188%: 188 − 184 = +4. It pays for about 12 buys and sells a year; holding pays for one buy.

Trades added up. A trade is one buy and the sell that ends it. Rule 4's account reinvests (it puts its whole value in at each buy), so its 53 trades compound to +159%: each trade starts with whatever the account now holds. Added up instead, as if each trade started again from $10,000, they come to +117%: +20% twice compounds to 1.2 × 1.2 = 1.44, +44%, but adds up to +40%. Against holding's compounded +184%, the lead becomes 117 − 184 = −67.

Two of the three turned a rule that trailed holding into one that seemed to beat it; the third made it look worse. Same market, same days, same costs, same reinvestment.

Less time in, less swing

On this market rule 1 held the market on 71% of the days and cash on the rest. Cash doesn't move, so rule 1's account had less swing (how far the account moves up and down) than holding's. Lesson B12's Sharpe ratio puts return per unit of swing on one scale: the higher it is, the more return for each unit of swing. Which is higher on this market, after costs?

Two Sharpe ratios with their error bars, one simulated market

Your guess:

Rule 1: 0.44. Holding: 0.76. But each is measured on one market's nine years, so each comes with an error bar about 0.64 either side: the range it could plausibly have landed in had luck gone differently. If either bar holds the other's number, the two can't be ranked, and here each bar holds the other's number. Across the 1,000 simulated markets the two could be ranked in only 18, after costs.

"Can't rank" is not "equal". Measured over all 1,000 markets' days together, where luck has little room left, holding's Sharpe ratio is the higher in this generator: 0.50 against rule 1's 0.34, after costs. Rule 1's days out remove more rise than swing, and it pays costs.

For the preciseAs lesson B12 works it: the yearly Sharpe ratio is (μ ÷ s) × √252, with μ the average daily return after costs (cash earns nothing here, so nothing is taken off for it) and s the standard deviation of the daily returns, dividing by n − 1. Rule 1: 0.0233 ÷ 0.842 = 0.0277, × 15.87 = 0.44. Holding: 0.0499 ÷ 1.047 = 0.0477, × 15.87 = 0.76. The error bar is the yearly Sharpe ratio ± 1.96 × √((1 + S² ÷ 2) ÷ n) × √252, with S the daily ratio and n = 2,348 days: 0.0277² ÷ 2 = 0.0004, 1.0004 ÷ 2,348 = 0.000426, √0.000426 = 0.0206, × 15.87 = 0.327, × 1.96 = 0.64, so rule 1 runs from −0.20 to 1.08 and holding (the same 0.64) from 0.12 to 1.40. The simulated market trades every weekday, about 261 days a year; the yearly figure keeps the usual 252, as lesson B12 does. The formula assumes independent days drawn from one bell curve; this simulation's days have fat tails and calm and wild spells, yet the bars held their long-run values in 954 (holding) and 932 (rule 1) of 1,000 markets, near the 950 a 95% bar should. Holding's Sharpe ratio was the higher in 805 markets. Rule 1 and holding share most of their days, so their Sharpe ratios rise and fall together: across the markets holding's had a standard deviation of 0.32 and the gap one of 0.21, which makes judging the gap by the two bars cautious.

One lead is one draw

Rule 4 trailed holding by 25 points on the lesson's market. Run it on other markets from the same generator, one at a time, then on all 1,000.

Rule 4's lead over just holding, after costs (simulated)

Each column counts the markets with a lead in a range 25 points wide. Dashed: a lead of 0. Colour: beat holding.

Markets run0
Latest lead–
Beat holding–

Across the 1,000 markets rule 4's lead ran from −569 to +123 points, and it beat holding in 155 of them, with nothing real to find. So a lead on one backtest is one draw: one measurement with an error bar (lesson B5). A held-out test is the next evidence (lesson B10).

The wrong benchmark

A benchmark can flatter. Count how often rule 1 "beat" each of three wrong benchmarks across the 1,000 markets, against how often it beat the right one, all after costs.

Rule 1 against four benchmarks, 1,000 simulated markets

Cash (a result above zero) flatters most: 752 against 138, because these markets rise on average and the rule holds them most days. Another market, held over the same days, brings its own luck into the comparison: rule 1 "beat" it in 318 markets. Other dates (holding from 2015) gave 124, close to 138, but not in the same markets: the extra year carries its own luck.

For the preciseAgainst holding from 2015 the verdict changed in 88 of the 1,000 markets, in both directions.

What this means for your backtest

  1. Ask "did it beat just holding?", not "did it make money?"

    In this simulation, where the markets rise on average and a rule out part of the time cannot pick its days, rule 1 beat holding in only 174 of 1,000 markets before costs and 138 after: such a rule tends to trail holding for that reason alone.

Practice 1 of 2

"It beat the market"

The situationAn imagined sales page: "Our rule made +62% from 2020 to 2024 after costs. The market made only +35%." The small print: the +35% is just holding the same market from 2019 to 2023. Just holding it from 2020 to 2024, after its one buy, made +80%.

What is the rule's lead over holding, like with like?

Hint: lead = net − hold, with holding over the same days as the rule.

The rule ran 2020–2024, so holding must too: lead = 62 − 80 = −18. The page's +35% covers other dates; against it the rule seemed 62 − 35 = 27 points ahead.
Use holding over the rule's own dates, 2020–2024: lead = 62 − 80 = −18, 18 points behind. The +35% covers 2019–2023, other dates, and made the rule look 27 points ahead.
Practice 2 of 2

A friend's rule

The situationA friend backtested a rule on one simulated market, 2016–2024, against just holding the same market over the same days, with the same costs and both accounts reinvesting. The rule led holding by +12 points after costs.

What does that show?

The comparison was fair, but it was one market. In this lesson's simulation rule 4 led holding, after costs, in 155 of 1,000 markets with nothing real to find.
Zero is the wrong benchmark: in a market that rises on average, most rules make money. Just holding over the same days is the right one, and the friend used it.
The comparison was like with like, so the lead is real on that market. It is still one measurement with luck in it; a held-out test is the next evidence.
✓

Lesson done

A result means little without a benchmark to measure it against. For a rule on one market, the natural one is just holding that market over the same days, with the same costs (holding pays for its one buy) and the same reinvestment; the lead is the rule's result minus holding's. In this lesson's simulation, on markets that rise on average with nothing real to find, rule 1 made money in 752 of 1,000 and beat holding in 138, both after costs: a rule out of the market part of the time, unable to pick its days, tends to trail holding there for that reason alone. Compare the Sharpe ratios too, each with its error bar, and read one lead as one draw.

Key ideas

Benchmark
a simple alternative that a rule's result is measured against
Just holding
buying the market at the start, never selling, paying for the one buy
Percentage points
the plain difference of two percents
Lead
the rule's result after costs minus just holding's on the same market over the same days, in percentage points

Formulas

lead = net − hold

The lead needs the same market, days, costs and reinvestment for both.

Lesson notes

A result means little without a benchmark, something to measure it against. For a rule on one market the natural one is just holding that market: buying at the start, never selling, and paying the cost of the one buy. The lead is the rule's result after costs minus holding's, over the same days, in percentage points. Measured like with like (same market, same days, same costs, same reinvestment), it answers "did it beat just holding?". Every count below is from the lesson's simulation, on markets built to rise slowly on average with nothing real for a rule to find.

"It made 59%" says nothing on its own

A backtest runs a trading rule over past prices to see what it would have made. On one simulated market, rule 1 held the market when its close was above the average of its last 200 closes and held cash otherwise, deciding at each close (a day's last price), trading at the next open (a day's first price) and paying 0.1% of the amount traded in costs on every buy and every sell (lesson A16). From 2016 to 2024 it turned $10,000 into $15,912, +59% after costs. Just holding the same market, bought at the first open of 2016 and never sold, turned $10,000 into $28,353, +184% after its one buy. So rule 1 was 184 − 59 = 125 percentage points behind: points are the plain difference of two percents.

Three more rules ran on the same market and days with the same costs. Rule 2 held when the close was above the average of its last 20 closes; rule 3 held on exactly the days rule 2 didn't; rule 4 held every day except the day after a close 2% or more below the close before. They made +53%, +7% and +159%. All four made money; none beat holding. Their leads were −125, −131, −177 and −25 points.

Making money is the easy part in a rising market

The four rules then ran on 1,000 more simulated markets over the same years, each drawn by the same generator (the computer recipe that drew the lesson's market). These markets rise slowly on average, and past prices say nothing about which way the next move goes; only the size of moves clusters, into calm and wild spells (runs of small moves and runs of big ones). So no rule had anything real to find. All counts here are after costs. Rule 1 made money (ended above $10,000 after costs) in 752 and beat holding in 138; rule 2, 615 and 59; rule 3, 541 and 40; rule 4, 853 and 155. Holding made money in 900.

On a day a rule holds cash it misses that day's move, and in markets that rise on average the missed days take part of the rise away. Even before costs rule 1 beat holding in only 174. The less a rule was in the market, the further it trailed. Averaged over the 1,000 markets, the rules held the market on 97% (rule 4), 63% (rule 1), 55% (rule 2) and 45% (rule 3) of the days, with average leads after costs of −37, −70, −110 and −123 points. Costs add to it: rules 2 and 3 bought or sold about 31 times a year. Rules 2 and 3 show the missed-days reason plainly: on every day exactly one of them holds the market, and on the lesson's market, before costs, rule 2 turned each $1 into $2.011 and rule 3 into $1.411. 2.011 × 1.411 = 2.838, exactly what holding turned $1 into before costs. They split holding's rise between them.

Like with like

On the lesson's market rule 4 trailed holding by 25 points. Three mismatches change that. Holding measured from the first open of 2015, a year before the backtest's first day (the rule needs a year of past closes before it can start), made +95%, because 2015 was a falling year on that market, and against it rule 4 seemed +64 points ahead. Rule 4 before costs made +188%, +4 points against holding after its one buy; rule 4 pays for about 12 buys and sells a year. A trade is one buy and the sell that ends it. Rule 4's account reinvests (it puts its whole value in at each buy), so its 53 trades compound to +159%, each starting with whatever the account then holds; added up instead, as if each started again from $10,000, they give +117%, −67 points against holding's compounded +184% (+20% twice compounds to 1.2 × 1.2 = 1.44, +44%, but adds up to +40%). Two mismatches flattered the rule, one hurt it. A fair comparison uses the same market, the same days, the same costs and the same reinvestment.

The swings

On the lesson's market rule 1 held the market on 71% of the days and cash on the rest, and cash doesn't move, so its account had less swing (how far the account moves up and down) than holding's. Lesson B12's Sharpe ratio puts return per unit of swing on one scale: after costs, rule 1's was 0.44 and holding's 0.76. Each is measured on one market's nine years, so each comes with an error bar about 0.64 either side: the range it could plausibly have landed in had luck gone differently. If either bar holds the other's number, the two can't be ranked, and here each bar holds the other's number. Across the 1,000 markets the two could be ranked in only 18, after costs. "Can't rank" is not "equal": over all 1,000 markets' days together holding's Sharpe ratio is the higher in this generator, 0.50 against rule 1's 0.34, after costs, because rule 1's days out remove more rise than swing and it pays costs.

What a benchmark can't do

A lead on one backtest is one draw. Across the 1,000 markets rule 4's lead ran from −569 to +123 points, and it beat holding after costs in 155, with nothing real to find. It is one measurement with an error bar (lesson B5), and a held-out test is the next evidence (lesson B10). The wrong benchmark flatters, too (counts after costs). Against cash, a result above zero, rule 1 "won" in 752 markets. Against holding another market over the same days it "won" in 318, because the other market's luck enters the comparison. Against holding the same market from 2015 it "won" in 124, close to 138, but the verdict changed in 88 markets, in both directions.

Formula: the lead over holding

The lead is the rule's result minus just holding's, in percentage points.

lead = net − hold
net
the rule's result after costs over the test days, in % = (end value ÷ start value − 1) × 100
hold
just holding's result on the same market over the same days, after its one buy, in % = (end value ÷ start value − 1) × 100
lead
percentage points ahead of holding; a minus sign means behind
Worked example: rule 4

Rule 4 made +159% after costs and holding +184% after its one buy: lead = 159 − 184 = −25, 25 points behind, although it made 159%.

Holds whenBreaks when

Any of the three differs. And a lead from one market is one draw, not the rule's lead on every market; the next evidence is a held-out test (running the rule once on days it was never tuned on: the Held-out test lesson).

One more worked example

Made-up numbers

Five years, 1,260 days, on one market. A rule's account went from $10,000 to $12,700 after costs; just holding, over the same days and after its one buy, from $10,000 to $14,200. net = 2,700 ÷ 10,000 = +27%, hold = +42%, lead = 27 − 42 = −15 points.

Common mistakes

Check yourself

  1. A rule's account went from $10,000 to $11,800 after costs. Just holding the same market over the same days went from $10,000 to $12,500 after its one buy. What is the lead?
    Answer

    net = 1,800 ÷ 10,000 = +18%, hold = +25%, lead = 18 − 25 = −7 points: 7 behind holding.

  2. A rule on one market ran from 2018 to 2024. Which benchmark is like with like: cash, holding another market over 2018–2024, or holding the same market over 2018–2024?
    Answer

    Holding the same market over 2018–2024, paying the same 0.1% cost for its one buy and reinvesting the same way. Cash flatters in a rising market, and another market brings its own luck into the comparison.

  3. Why can rule 2 and rule 3 not both beat holding on the same market before costs?
    Answer

    On every day exactly one of them holds the market, so before costs their growth multiplies to holding's: on the lesson's market 2.011 × 1.411 = 2.838. If one turns $1 into more than holding does, the other turns it into less than $1, and if both made money, both trail holding.

  4. Why did rule 1's account swing less than holding's on the lesson's market?
    Answer

    It held cash, which doesn't move, on the 29% of the days it was out of the market (it was in on 71%). Its Sharpe ratio, return per unit of swing, was still lower: 0.44 against holding's 0.76, after costs, and the error bars couldn't rank the two.

Try it yourself

Run the script. Every number in this lesson comes from one Python script (below). It imports the course's shared simulated market generator, simmarket.py, from the same folder, builds the lesson's market and 1,000 more, runs the four rules and just holding on each, and counts. Change a rule, the cost, the years, or switch the market's planted effect back on, and see what moves.

By hand, in a spreadsheet. Put the date in column A, the rule's account value at each close in column B and just holding's in column C, both starting at 10,000 on the same day and both after costs. The results are =B_last/B_first-1 and =C_last/C_first-1, and the lead is their difference, in points. For the Sharpe ratios, put each day's return in columns D and E (=B3/B2-1), then =AVERAGE(D:D)/STDEV.S(D:D)*SQRT(252) for each.

The script needs simmarket.py, the course's simulated market, in the same folder.

The code behind every number (Python, on the course's simulated-market generator; market seeds 2026141300 to 2026142300)
Loading…

Going further: the gap itself, and the other rules

Why bars overlap while the gap is real. Rule 1 and holding are measured on the same days, so their Sharpe ratios rise and fall together: a good year for the market is a good year for both. Across the 1,000 markets holding's Sharpe ratio had a standard deviation of 0.32, while rule 1's minus holding's had one of 0.21. The gap between the two moves less than either number, so judging it by the two bars is cautious: it can miss a real gap. A bar for the gap itself needs how closely the two move together, which lesson B12 does not cover. For rule 4, in the market on 97% of days, the gap's standard deviation was 0.10.

The other rules, by Sharpe ratio. Over all the markets' days, rule 4's Sharpe ratio was 0.42, rule 2's 0.16 and rule 3's 0.08, against holding's 0.50. A rule in the market on a share f of days chosen with no skill keeps about f of holding's average day but about √f of its volatility, so its Sharpe ratio is about √f times holding's before costs: rule 1, in on 63% of days on average, √0.63 × 0.50 = 0.40, and costs take it to 0.34. Rule 4 was ahead of holding by Sharpe ratio in 182 markets and by result in 155; rule 2 in 74 and 59; rule 3 in 43 and 40. The bars ranked rule 2 against holding in 115 markets and rule 3 in 185, always with holding higher.

Before costs. Rule 1 beat holding before costs in 174 markets, rule 2 in 196, rule 3 in 177 and rule 4 in 429. Rule 4 is in the market almost every day, so before costs it is close to holding and beats it often; its costs, about 16 buys and sells a year on average, are most of its gap.

The wrong benchmarks, for every rule. Against holding another market the verdict changed in 376 markets for rule 1, 228 for rule 2, 168 for rule 3 and 439 for rule 4; against holding from 2015, in 88, 44, 41 and 169.

Honest notes

Where this comes from