Quant from Scratch by QuantWright

Backtesting Essentials · Lesson 14

Robustness

Does a result survive nearby settings, other markets and reshuffled trades? What each check shows, and what none of them can.

Know the basics? Read the lesson notes

One rule, one setting

A backtest runs a trading rule over past prices to see what it would have made. Here the prices are simulated, 2015 to 2024, by a computer program, the generator. Each evening the market publishes a made-up number, the reading, and a reading of +1 adds 0.10% to the next day's move, about a tenth of a typical day. We built that in, so it is real.

The rule: if today's reading is above k, be in the market tomorrow, with $100 in at the day's first price and out at its last, before costs. A rule's lead is its result minus just holding's (in every day) over the same days; divided by the five years, it is the lead a year.

Slide k through its seven values; the leads are for 2015 to 2019.

One setting, seven values: lead a year, 2015–2019 (simulated)

Each cell is one value of k. Colour: led just holding. Grey: trailed it. Pale: $0.

Values seen1 of 7
Lead a year$0

The leads run from −$9 to $2 a year. Just holding made $53 over these years, and a higher k is in on fewer days, so it misses more of that rise.

A second setting, a grid

Now a second setting, L: be in tomorrow if the average of the last L readings is above k (L = 1 is today's alone). Seven values of each make 7 × 7 = 49 versions, each one full set of settings (the Overfitting lesson), laid out here as a grid.

2015 to 2019 are the choosing years: the grid is drawn and a version picked on them. 2020 to 2024 are held out for one run at the end (the Held-out test lesson). Sweep L from 1 to 7.

49 versions: lead a year, 2015–2019 (simulated)

Rows: L. Columns: k. Colour: led just holding. Grey: trailed it. Pale: $0.

The left-hand columns, k = −1.5 and −1.0, are flat at $0 to $2: so low a k is in almost every day, close to just holding. The right side, out on most days of a rising market, trails. The grid shows whole dollars; three versions show its highest, $2, and the first of them, L = 1 and k = 0.5, sits beside −$5 and above −$4.

Two versions are neighbours when they are one step apart in either setting or both, diagonals included. Neighbours are in or out on mostly the same days, so they share those days' luck: luck lands on a grid in patches, not cell by cell.

For the preciseAveraged over all 156 neighbouring pairs in this grid, two neighbours were in or out together on 85.5% of the choosing days.

A peak or a plateau?

A lone high version among lower ones is a peak; a wide patch of similar results is a plateau. A common rule of thumb says trust the middle of a plateau, not the peak: a real effect should survive nearby settings, while luck can light up one version.

The plateau middle is the version whose average with its neighbours is highest (the next card works it out). Marked: A, the peak (the first of the three at $2); B, the plateau middle; C, a version drawn at random. Which would you trust on the held-out years?

49 versions: lead a year, 2015–2019 (simulated)

A: the peak. B: the plateau middle, its neighbours dashed. C: drawn at random.

You'd trust:
A, held out?
B, held out?
C, held out?

On 2020 to 2024 A led by $12 a year, B by $0 and C by $8. Just holding lost $38 there, so being out of the market paid by itself: out every day, 38 ÷ 5 = $7.60 a year. B was in on 1,293 of the 1,295 choosing days, almost just holding.

One market is one draw, and this one favoured the peak. Next, 1,000 markets.

For the preciseThree versions reached $2 a year on the choosing years: L = 1 with k = 0.5, and L = 3 and 4 with k = −1.0. The peak is the first in order, smallest L first. Leads a year are rounded to whole dollars, halves up, and the picks are made from those rounded figures. To the cent, B's held-out lead was $1.30 over the five years. C was drawn at random from the 47 versions that are neither A nor B. A was in on 433 of the choosing days.
Theory

The neighbourhood average

Checking how a result changes as the settings move is a sensitivity analysis. One simple score is a version's neighbourhood average: the average of its own result and its neighbours', the versions one step away in either setting, or in both. The plateau middle is the version with the highest.

N = (R1 + R2 + … + Rm) ÷ m
R1 … Rm
the results of a version and of every version next to it in the grid (leads a year, in this lesson)
m
how many of those are in the grid: 9 for a version inside, 6 on an edge, 4 in a corner
N
the version's neighbourhood average, for versions all run on the same days
Worked example: a made-up grid

Two settings with four values each. Row by row: 0, 2, −4, 16; then 3, 7, 6, −5; then 4, 8, 7, −2; then 1, 3, 2, 0.

Solid: the peak, 16, and the plateau middle, 8. Dashed: each one's neighbours.

The peak, 16, is in a corner, so m = 4: (16 − 4 + 6 − 5) ÷ 4 = 13 ÷ 4 = 3.25. The 8 in the third row is inside, so m = 9: (3 + 7 + 6 + 4 + 8 + 7 + 1 + 3 + 2) ÷ 9 = 41 ÷ 9 = 4.56, the highest of all 16 neighbourhood averages: the plateau middle.

On your grid: B, on an edge, (1 + 1 + 0 + 2 + 0 + 2) ÷ 6 = 1.00. A, (1 + 2 − 5 − 5 − 4 − 6) ÷ 6 = −17 ÷ 6 = −2.83.

Holds when
  • the steps between settings are small, so neighbours are near copies;
  • every version was run on the same days;
  • the grid was fixed before any result was seen.
Breaks when

The grid is wide and searched: the best neighbourhood is then the best of many too (the Best of many lesson). A version on an edge or in a corner averages fewer results, so luck moves it more. A real effect can be narrow, and averaging then blurs it.

New numbers. A grid has the rows 4, 6, 3; then 5, 9, 2; then 1, 3, −4. What is the neighbourhood average of the 6, in the middle of the top row?

The 6 is on the top edge, so m = 6: the 4 and 3 beside it, and the 5, 9 and 2 below. (4 + 6 + 3 + 5 + 9 + 2) ÷ 6 = 29 ÷ 6 = 4.83.

1,000 markets with the effect, 1,000 without

We built 1,000 more markets this way, and the same 1,000 again with the effect switched off. Each of these is a twin: the same draws without the extra push, so nothing in it can be found. On each, take the peak and the plateau middle from 2015–2019 and run each once on 2020–2024. A pick holds up if its held-out lead is above $0, even by a cent. A blind draw takes a version at random; its count is an expected one: the share of the 49 versions that held up, times 1,000.

Guess first: in the markets with the effect, which held up more often?

Picks that held up on 2020–2024, of 1,000 markets (simulated)

Your guess:
Markets:

With the effect, the peak held up in 802 markets, the plateau middle in 795 and a blind draw in about 439. A gap of 7 in 1,000 is well within chance.

So in this simulation the plateau middle did no better than the peak. The grid was smooth already: the plateau middle was the peak or a neighbour of it in 847 of 1,000. Both beat a blind draw because the reading works across a broad patch of settings.

Without the effect: 326, 264 (the plateau middle often settled on a version in the market every day, whose lead is exactly $0) and about 245. These markets rise on average, so a blind draw that is out on most days trails; the picks beat it for that reason, not because they found anything.

For the preciseWith the effect the plateau middle was the peak in 291 markets; of the 709 where they differed, the plateau middle did better on the held-out years in 364 and the peak in 345. Without the effect the plateau middle's held-out lead was exactly $0 in 248 markets: a version in on every held-out day is just holding, and $0 is not above $0. Held-out leads a year, averaged over the 1,000 markets with the effect: the peak $3.35, the plateau middle $3.41, all 49 versions −$1.69. Without: −$1.79, −$1.36 and −$3.81. Without the effect the plateau middle did better than the peak in 523 markets and worse in 386. A blind draw's count is the average share of the 49 versions with a held-out lead above $0, times 1,000. Counts compare exact cents; all results are before costs.

Other markets

A real effect should also show on markets it was not chosen on. The generator makes five independent markets at a time; so far everything ran on Market 1. Run A, B and C unchanged on Markets 2 to 5, over the same years.

A, B and C on four other markets: lead a year, 2015–2019 (simulated)

Rows: the three picks from Market 1. Columns: Markets 2 to 5.

A led on 3 of the 4, B on all 4, by $0.25 to $1.35 a year (it is in on almost every day, so two of those leads round to $0 in the grid), and C on 1.

Across the 1,000 markets with the effect, the plateau middle led on 3 or 4 of its four other markets in 796; without the effect, in 103.

So here the check told the two kinds of market apart, because every market of this generator carries the same effect and they move independently. Real markets need not share an effect, and often move together, so luck can show on several at once.

For the preciseOf the 796, 676 held up on the held-out years; of the 103, 46. The peak led on 3 or 4 of its other four markets in 777 of the 1,000 markets with the effect and 138 without; a blind draw in about 318 and 70. To the cent, B led on Markets 2 to 5 by $2.25, $6.75, $5.89 and $1.25 over the five years.

Reshuffled trades

The Monte Carlo lesson's check: put a backtest's trades in random orders. A's 433 trades on the choosing years (one per day it was in) are reshuffled 1,000 times; drawn together, the paths make a fan. Then switch to the twin market: the same version on the same 433 days.

A's trades in 30 of 1,000 random orders (simulated)

The result so far, from $0. Grey: reshuffles. Colour: A's own order.

Every order ends at–
Own maximum drawdown–
Reshuffles' 95th percentile–

Every reshuffle ended where its backtest ended, $62 with the effect and $23 without: a sum is the same in any order. Nothing in either fan shows which result is real; a lucky list ends high in a tidy fan too.

A fan shows the path, such as the maximum drawdown: the deepest fall below the highest point so far. Its 95th percentile is the drawdown only 5 in 100 reshuffles pass, so chance alone puts a backtest's own beyond it about 50 times in 1,000. Over the 1,000 markets the peak's went beyond it 68 times with the effect and 62 without: about the same. A reshuffle checks the ride, not whether the result is real.

For the preciseThe 95th percentile is the 950th of the 1,000 sorted maximum drawdowns (the Monte Carlo lesson). With the effect A's own maximum drawdown was $12 and the 95th percentile $17; without the effect A's own maximum drawdown was $13 and the 95th percentile $24. Both counts sit a little above the 50 that independent days would give, because this generator's days come in calm and wild spells, which an order keeps and a reshuffle breaks. For the plateau middle the counts were 66 and 61 (without the effect 22 plateau middles were never in, so had no trades).

What these checks can't do

None of these checks is a held-out test, one run of a chosen rule on years set aside first (the Held-out test lesson). Three limits, two of them in counts:

Two limits, over 1,000 markets of each kind (simulated)

A plateau of luck. Without the effect the best neighbourhood average on the choosing years was still above $0, by any amount, in 725 of 1,000 markets (976 with it). Near copies share luck, so a plateau shows where versions overlap, not that something is real.

Chosen after looking. The plateau middle is the best of 49 neighbourhood averages, so it is still the best of many (the Best of many lesson), flattered like any other.

Short data. The version with the highest average held-out lead over the 1,000 markets with the effect, L = 1 and k = −0.5 ($4.89 a year), led (a lead above $0) on its five choosing years in 900 of them, and on 2019 alone in 710. On that one year it trailed or only matched just holding in 1,000 − 710 = 290: on a short stretch a small real effect is easy to miss.

For the preciseWith the effect, the plateau middle's neighbourhood average on the choosing years averaged $4.25 a year, and its held-out lead $3.41. The peak's lead went from $7.36 a year on the choosing years, on average, to $3.35 on the held-out ones. On the lesson's market the version L = 1, k = −0.5 trailed by $1 a year on the choosing years and led by $11 on the held-out ones.

What this means for your backtest

  1. A peak and a plateau can both be luck.

    Neighbouring versions are near copies, so here picking either did about as well. Only the held-out test tells.

Practice 1 of 3

"My setting is the 18"

The situationA friend sweeps two settings of a rule and shows you the grid of leads a year, row by row: −3, 2, 1, 0; then 4, 1, 18, −5; then 0, 3, −2, 1. "The 18 is my setting."

What is the neighbourhood average of that version?

Hint: the 18 is inside the grid, so m = 9: three versions in the row above, three in its own row, three below.

(2 + 1 + 0 + 1 + 18 − 5 + 3 − 2 + 1) ÷ 9 = 19 ÷ 9 = 2.11. The 18 is a lone peak. That alone shows neither luck nor a real effect: in the lesson, picking a peak or a plateau middle did about as well. Only a held-out test tells.
Practice 2 of 3

"A plateau can't be luck"

The situationAn imagined sales page says: "We tested 400 settings and chose the middle of the widest plateau. A plateau can't be luck."

What is wrong with the last sentence?

Neighbouring settings are in or out on mostly the same days, so one lucky stretch lifts the whole patch: in the lesson, without the effect, the best neighbourhood average was above $0 in 725 of 1,000 markets.
No size makes a plateau real: neighbours share their days and so their luck. Only a held-out test tells.
Near copies share their luck: in the lesson, without the effect, the best neighbourhood average was above $0 in 725 of 1,000 markets. And the widest of 400 settings is still the best of many (the Best of many lesson).
Practice 3 of 3

Two checks of your own rule

The situationYour own rule, picked on one market. You reshuffled its trades 1,000 times, and you ran it, unchanged, on three other markets you never looked at while choosing.

Which check can add evidence that the effect is real?

They are new data for this rule: evidence, not proof, as real markets can move together. The reshuffles all end at the backtest's own result, so they say nothing about the edge.
Every reshuffle ends where the backtest ends, so a lucky list ends in profit 1,000 times too. The other markets, never used to choose, are new data.
True of the reshuffles, which only reorder the backtest's trades. The other markets played no part in the choice, so they are new data: evidence, not proof.
✓

Lesson done

A robustness check asks whether a result survives nearby settings, other markets and a new order of its trades. A lone peak and a plateau can both be luck, because neighbouring versions share most of their days and so their luck. In this lesson's simulation picking the middle of a plateau held up on held-out years about as often as picking the peak (795 against 802 of 1,000 markets with a small planted effect, a gap within chance; before costs), and a reshuffle checked only the path. None of these checks replaces the held-out test.

Key ideas

Version
one full set of a rule's settings
Grid
the results of every version laid out by their settings
Neighbours
two versions one step apart in either setting or both, diagonals included: near copies that share most of their days
Peak
a lone high version with lower ones around it; it can be luck, and so can a plateau
Plateau
a wide patch of similar results; neighbouring versions share most of their days, so a plateau can be luck too
Plateau middle
the version with the highest neighbourhood average; still the best of many versions

Formula

N = (R1 + R2 + … + Rm) ÷ m

The neighbourhood average of a version: its result and its neighbours', with m = 9 inside a grid, 6 on an edge, 4 in a corner. It compares versions run on the same days, in a grid fixed before looking; an edge or corner averages fewer results, so luck moves it more.

Lesson notes

A robustness check asks whether a backtest's result survives small changes: nearby settings, other markets, a new order of its trades. A lone high setting and a wide plateau can both be luck, because neighbouring settings share most of their days and so their luck. In the lesson's simulation, picking the middle of a plateau held up on held-out years about as often as picking the peak (a gap within chance), and only a held-out test tells. Every count below is from that simulation, before costs.

One rule, two settings

A backtest runs a trading rule over past prices to see what it would have made. The lesson's prices are simulated, 2015 to 2024, made up by a computer program, the generator. Each evening the market publishes a made-up number, the reading; a reading of +1 adds 0.10% to the next day's move, about a tenth of a typical day's size: a small real effect, built in. The rule is in the market the next day if the average of the last L readings is above k ($100 in at the day's first price, out at its last, no costs). Its lead is its result minus just holding's over the same days; the lead a year divides by five. Seven values of L (1 to 7) and of k (−1.5 to 1.5) make 49 versions, each one full set of settings, and their results laid out by setting form a grid. It is drawn on the choosing years, 2015 to 2019; 2020 to 2024 are held out, set aside first for one run at the end.

Two versions are neighbours when they are one step apart in either setting or both, diagonals included. On the lesson's market, neighbouring versions were in or out together on 85.5% of the choosing days, averaged over the grid's 156 neighbouring pairs. Near copies share their days' luck, so luck lands on a grid in patches.

A peak, a plateau and a blind draw

A peak is a lone high version; a plateau is a wide patch of similar results. The rule of thumb tested: trust the middle of a plateau, not the peak. On the lesson's market the peak, A, made $2 a year at L = 1, k = 0.5 (the first of three versions at $2 in whole dollars); the plateau middle, B, the version whose average with its one-step neighbours is highest (its neighbourhood average, worked out below: $1.00), sat at L = 3, k = −1.5; C was a blind draw, a version taken at random. On the held-out years A led by $12 a year, B by $0 and C by $8. Just holding lost $38 there, so a version out on many days led for that alone (out every day: 38 ÷ 5 = $7.60 a year), and B, in on 1,293 of the 1,295 choosing days, was almost just holding. One market is one draw.

1,000 markets with the effect, 1,000 without

The same study ran on 1,000 more markets, and on their twins: the same 1,000 with the effect switched off, so that nothing could be found. A pick holds up if its held-out lead is above $0, even by a cent. With the effect the peak held up in 802, the plateau middle in 795 and a blind draw in about 439 (an expected count: the average share of the 49 versions that held up, times 1,000). A gap of 7 in 1,000 is well within chance: in this simulation picking either did about as well, because neighbours are near copies and the grid was smooth already. Without the effect: 326, 264 (the plateau middle often settled on a version in the market every day) and about 245. These markets rise on average, so a blind draw that is out on most days trails, and the picks beat it for that reason alone; the plateau middle's held-out lead was exactly $0, not above it, in 248, where it was in every held-out day.

Other markets and reshuffled trades

The generator makes five independent markets at a time. Run unchanged on the other four over the choosing years, the plateau middle led on 3 or 4 of them in 796 of the 1,000 markets with the effect and in 103 without. It worked here because every market of the generator carries the same effect and they move independently; real markets need not share an effect, and often move together.

A reshuffle (the Monte Carlo lesson) puts a backtest's trades in a new random order, and a sum is the same in any order. A's 433 trades always ended at $62; the same days in the twin market without the effect always ended at $23. Plotting all 1,000 reshuffled paths together gives a fan, which shows the path, such as the maximum drawdown, the deepest fall below the highest point so far. Over the 1,000 markets the peak's own maximum drawdown went beyond what 95 of every 100 reshuffles reached (the fan's 95th percentile) in 68 with the effect and 62 without, where chance alone gives about 50: about the same either way.

What these checks can't do

None of them is a held-out test, one run of a chosen rule on years set aside first. A plateau of luck: without the effect, the best neighbourhood average on the choosing years was still above $0, by any amount, in 725 of 1,000 markets (976 with it). Chosen after looking: the plateau middle is the best of 49 neighbourhood averages, still the best of many; with the effect its neighbourhood average averaged $4.25 a year on the choosing years and its held-out lead $3.41. Short data: the version with the highest held-out lead on average over the 1,000 markets with the effect (L = 1, k = −0.5, $4.89 a year) led on its five choosing years in 900 of them and on 2019 alone in 710, so on that single year it trailed or only matched just holding in 290: on a short stretch a small real effect is easy to miss.

Formula: the neighbourhood average

Checking how a result changes as the settings move is a sensitivity analysis. A version's neighbourhood average is the average of its own result and its neighbours', the versions one step away in either setting or both; the plateau middle has the highest.

N = (R1 + R2 + … + Rm) ÷ m
R1 … Rm
the results of a version and of every version next to it in the grid (leads a year, in the lesson)
m
how many of those are in the grid: 9 for a version inside, 6 on an edge, 4 in a corner
N
the version's neighbourhood average, for versions all run on the same days
Worked example: a made-up grid

Rows 0, 2, −4, 16; then 3, 7, 6, −5; then 4, 8, 7, −2; then 1, 3, 2, 0. The peak, 16, is in a corner (m = 4): (16 − 4 + 6 − 5) ÷ 4 = 13 ÷ 4 = 3.25. The 8 in the third row is inside (m = 9): (3 + 7 + 6 + 4 + 8 + 7 + 1 + 3 + 2) ÷ 9 = 41 ÷ 9 = 4.56, the highest of the 16: the plateau middle.

Holds whenBreaks when

The grid is wide and searched, so the best neighbourhood is the best of many too (the Best of many lesson). An edge or corner averages fewer results, so luck moves it more. A real effect can be narrow, and averaging blurs it.

One more worked example

Made-up numbers

A grid of three rows: −2, 1, 0, 3; then 11, 2, 6, 5; then −3, 4, 7, 6. The peak, 11, sits on the left edge (m = 6): (−2 + 1 + 11 + 2 − 3 + 4) ÷ 6 = 13 ÷ 6 = 2.17. The 6 in the middle row is inside (m = 9): (1 + 0 + 3 + 2 + 6 + 5 + 4 + 7 + 6) ÷ 9 = 34 ÷ 9 = 3.78. The 6 in the bottom-right corner (m = 4): (6 + 5 + 7 + 6) ÷ 4 = 24 ÷ 4 = 6.00, the highest of the twelve, so it is the plateau middle. A corner averages only four results, so luck moves its score more than a middle version's nine.

Common mistakes

Check yourself

  1. A grid has the rows 5, 1, 2; then 3, 4, 6; then 0, 2, 8. What is the neighbourhood average of the 8, in the bottom-right corner?
    Answer

    A corner, so m = 4: (4 + 6 + 2 + 8) ÷ 4 = 20 ÷ 4 = 5.00.

  2. In the lesson's simulation, did picking the middle of a plateau hold up on held-out years more often than picking the peak?
    Answer

    No. With the small planted effect the plateau middle held up in 795 of 1,000 markets and the peak in 802 (a blind draw: about 439); without it, 264 and 326 (about 245). Neighbouring versions share most of their days, so the grid was smooth already and the peak usually sat on a plateau too. Those counts are this simulation's, before costs.

  3. Why can't a reshuffle of a backtest's trades show that its result is real?
    Answer

    A reshuffle only changes the order, and a sum is the same in any order, so every reshuffle ends at the backtest's own result. A lucky list ends high too. A reshuffle measures the path, such as the maximum drawdown.

  4. When does running a rule on other markets add evidence, and why is it weaker on real markets than in the lesson?
    Answer

    When those markets played no part in choosing the rule. In the lesson every market carried the same effect and moved independently; real markets need not share an effect, and they often move together, so luck can show on several at once.

Try it yourself

Run the script. Every number in this lesson comes from one Python script (below), which imports the course's shared simulated market, SimMarket. It builds 1,000 sets of five markets with the effect and the same 1,000 without, draws each grid, takes the peak and the plateau middle, and counts. Change the grid's values, the stretches, or the neighbourhood (two steps instead of one), and see what moves.

By hand, in a spreadsheet. Put a grid of results in a block of cells, say B2:H8. In another block, the neighbourhood average of the cell in C3 is =AVERAGE(B2:D4); on an edge or a corner, shrink the range so it stays inside the grid. The largest of these is the plateau middle.

The code behind every number (Python; imports the course's simulated-market generator, simmarket.py; seed 2026111400, market seeds 2026111401 to 2026112400)
Loading…

Honest notes

Where this comes from