Backtesting Essentials · Lesson 10
The held-out test
The last test of a rule: years nothing has touched, one run, and the result written down whatever it is.
Know the basics? Read the lesson notesSet two years aside before anything else
A backtest runs a trading rule over past prices to see what it would have made. Here the prices are simulated: ten years, 2015 to 2024, made up by a computer. You stand at the end of 2022. Eight years exist; 2023 and 2024 have not happened.
A result flatters a rule once it has helped to shape or choose it (lessons B7 to B9), so the last test of a rule needs years that nothing has touched. Years set aside for it are held out, and they are set aside first: a year you have looked at has already shaped what you try. The rest are the working years.
Which two years would you set aside?
Ten simulated years (the last two have not happened)
Dark: to work on. Hatched: held out. Dashed: not happened yet.
You chose the first two years. The lesson sets aside the last two, 2021–2022.
You chose two years from the middle. The lesson sets aside the last two, 2021–2022.
You chose the last two years, 2021–2022, and so does the lesson.
A rule is built on the past and traded after it. The last two years copy that: everything the rule was built on comes before them, and nothing it was built on comes after.
From now on nothing looks at 2021–2022 until the one run.
The yardstick: just holding
Each day a rule is in the market or out of it. In: $100 goes in at the day's first price and comes out at its last. A rule's result is its days added up. No trading costs are taken off, so every result in this lesson is before costs.
Just holding is in every day. A rule's lead is its result minus just holding's over the same days. Divided by the number of years, it is the lead a year.
Over two years a rule makes $40 and just holding makes $30. What is the rule's lead a year?
All the work happens on the other six years
The six working years are 2015 to 2020. Each evening this market publishes a made-up number, the reading, and the next day's move tends to go a little its way. We built that in, so it is real. Nothing else below is.
A rule sorts days into kinds by conditions: was yesterday's reading above 0, which month is it, which weekday, did yesterday's price rise? Tuning gives each kind of day a setting, in or out: in, if that kind's days added up to a gain on 2015–2018. Every combination of the four conditions with at least one in use is a rule: 15 rules. Eight use the reading. The other seven have nothing real in them.
All 15 then run on 2019–2020, settings unchanged. The one that makes the most there is the pick.
15 tuned rules on 2019–2020 (simulated)
Dark: tuned on. Grey: picked on. Hatched: held out. Dots: each rule's lead a year.
The pick is the rule with the reading alone. On 2019–2020 it made $54 and just holding $32: a lead of 54 − 32 = $22, or 22 ÷ 2 = $11 a year. Rule 7 tied with it to the cent (its settings came out the same); the pick is the first of the two.
That is not a fair report: $11 a year is the best of 15 on the years that picked it, and the best of many tries is partly luck (lesson B7). On 1,000 more markets built the same way, the pick's lead on 2019–2020 averaged $7.66 a year.
For the precise
Each of the four conditions is used or left out, which makes 2 × 2 × 2 × 2 = 16 combinations, less the one that uses none: 15 rules, with from 2 to 240 settings. To one decimal the pick made $54.0 on 2019–2020 and just holding $32.5. Whole-dollar figures round halves up, and on the market shown every lead is taken between the whole-dollar figures. Rule 7 sorts days by the reading and by yesterday's price; tuning set it to be in the market on the same days as the reading alone. The tuning years hold 1,039 trading days, 2019–2020 523, the held-out years 521 and 2023–2024 522. On the 1,000 other markets the pick used the reading in 727.Write the plan, then run it once
Before anyone looks at the held-out years, write the plan down: the rule, the years, one run, and the line the rule must clear. Here the line is a lead above $0: clear it and the rule passes, otherwise it fails.
The plan comes first: a line drawn after the result can be drawn to fit it. The result is recorded whatever it is. This one run is the held-out test. Will the pick pass?
The plan, and the one run (simulated)
Colour: the pick. Grey: just holding.
You guessed a pass, and it passed.
You guessed a fail. It passed.
On 2021–2022 the pick made $24 and just holding lost $1: a lead of 24 + 1 = $25 before costs, or 25 ÷ 2 = $12.50 a year. That is above $0, so the pick passes, and the one run is used.
That is one market. On the 1,000 others the pick passed in 505 and failed in 495, and its held-out lead averaged $0.78 a year, against $7.66 on the years that picked it.
For the precise
To one decimal the pick made $23.7 on 2021–2022 and just holding −$1.0. Of the 495 fails, the pick trailed just holding in 440; in the other 55 its tuned settings were in the market every day, as just holding is, so its lead was exactly $0, which is not above the line. The pick's lead a year was lower on the held-out years than on 2019–2020 in 735 of the 1,000 markets. Counts over the 1,000 markets compare exact cents, and averages over them are given to the cent.One result is one draw
A lead over two years has luck in it. To see how much, run the same study on other markets: the same plan, one run each.
The pick's held-out lead a year on each market (simulated)
Colour: passed. Grey: failed. Diamond: the market shown.
The bars count the 1,000 picks by their held-out lead a year. The markets were built the same way, so the spread is mostly luck. The middle 950 of the 1,000 leads (the 25 highest and the 25 lowest left out) run from −$14.40 to $17.30 a year, around the average of $0.78. The $12.50 of the market shown sits inside that range.
The rule with the reading alone is real, and run on the held-out years of every market it still failed in 455 of the 1,000. (The pick's 495 fails are a different count: the pick is whichever rule won on 2019–2020 in that market, the reading alone in only 288.) The rule with the month alone has nothing in it, and it passed in 339.
So a pass is evidence, not proof, and a fail is not proof either. What the held-out test adds is one result that no tuning or picking has touched.
For the precise
The lowest of the 1,000 leads was −$23.90 a year and the highest $27.10; single markets' leads a year are given to the nearest 10 cents. The middle 950 leave out the 25 lowest and the 25 highest. The rule with the reading alone passed in 545 markets, and its held-out lead averaged $3.18 a year.A second look spends it
The plan said one run. The tempting next move: run the other 14 rules on 2021–2022 as well, and keep the best.
All 15 rules on the held-out years (simulated)
Dots: each rule's lead a year on 2021–2022. Dark ring: the pick. Colour: the best of 15.
On this market 12 of the 15 rules passed, six of them with nothing real in them: on this market a rule needed nothing real to pass. The best of 15, the reading with the weekday, made $33 where just holding lost $1: a lead of 33 + 1 = $34, or 34 ÷ 2 = $17 a year, above the pick's $12.50.
But 2021–2022 have now picked a rule. Lesson B9's word for data that has tuned a rule or picked it is in-sample, and a result on in-sample data flatters the rule. For the best of 15 the held-out years are now in-sample, just as the working years are for the pick.
The simulation can let 2023 and 2024 happen. There the best of 15 made $11 and just holding $35: a lead of 11 − 35 = −$24, or −$12 a year.
That is one market. On the 1,000 others the best of 15 passed in 892, and its lead averaged $7.71 a year on the held-out years and $0.46 on the two after. The pick, run once, averaged $0.78 and then $0.77: on average, a fair report of what came next.
A second look, a changed setting, a second rule: each lets the held-out years tune or pick, and the fair report is gone.
For the precise
To one decimal the best of 15 made $33.2 on 2021–2022 and $11.4 on 2023–2024, where just holding made $35.4. On 2019–2020 it had ranked 3 of the 15. On the 1,000 other markets the best of 15 was the same rule as the pick in 150, and its lead a year was lower on the two years after than on the held-out years in 750. For the pick that count was 462, with 483 higher and the rest level. The $7.71 is close to the pick's $7.66 on 2019–2020 for a reason: each is the best of the same 15 rules on two years.After the test
The pick passed. What is a pass worth? Let 2023 and 2024 happen on the 1,000 other markets. Guess first: of every 100 picks that passed, how many led just holding over those two years?
Who led over 2023–2024? (simulated, 1,000 markets)
You guessed about 55, the closest of the three. It was 57.
You guessed about 75. It was 57.
You guessed about 95. It was 57.
Of the 505 picks that passed, 290 led just holding over 2023–2024: 57 in every 100. Of the 495 that failed, 215 did: 43 in every 100. Of all 1,000 picks, about half led. So in this simulation a pass is evidence, and far from proof: the reading's effect is small and two years are short. A gap of 57 against 43 is a small one.
A pass sends the rule on to a forward test: the rule, unchanged, run on new days as they arrive (lesson B16). That is one more check, not a green light. On the market shown the pick passed and then trailed: $28 against just holding's $35, a lead of 28 − 35 = −$7.
A fail ends this test of this rule. It does not prove the rule worthless, but running it again would be a second look, which makes the years in-sample. A new idea needs new held-out data, or the wait for new data to arrive.
The error bar on a held-out lead
A held-out lead comes from a limited number of days, so luck moves it. Lesson B5 measured that: the standard error is how far luck typically moves a measured number, and the 95% confidence interval, or error bar, reaches 1.96 standard errors either side. A lead added up over n days has a standard error of one day's spread times √n.
- lead
- a rule's result minus just holding's over all the held-out days: the total, not the lead a year
- σ
- the standard deviation of one day's lead, measured in the held-out run itself: how far a day's lead typically sits from the average day's (σ is the Greek letter sigma)
- n
- the number of held-out days, so σ × √n is the standard error of the lead
- 1.96
- standard errors either side for a 95% interval: 95 in 100 such intervals catch the true value, which is the rule's real long-run lead
The pick's day-by-day leads over the n = 521 held-out days had σ = $0.74. √521 = 22.8, so the standard error is 0.74 × 22.8 = 16.9 and the error bar 1.96 × 16.9 = 33.1, about $33 either side of the lead. The lead here is the two-year total, $25 (twice the $12.50 a year): from 25 − 33 = −$8 to 25 + 33 = $58.
Zero is inside, so this pass cannot rule out a rule with no lead. Of the 505 passes on the other markets, zero was inside the bar in 472.
A real lead is one the rule keeps earning, on average, day after day. Four times the days give four times a real lead and only twice the bar (√4 = 2). So a short held-out stretch leaves the bar wide next to the lead.
- one rule, one run, as planned;
- one day's lead does not change the next day's;
- there are hundreds of days, not dozens.
The lead is the best of several runs on the held-out days: the bar then sits around a flattered number. It also breaks when the market changes under the rule (lesson C3, Regimes and decay).
New numbers. A held-out lead is $30 over 400 days, and one day's lead has σ = $0.90. How many dollars either side does the error bar reach?
What this means for your backtest
- Set the last stretch aside first.
Before tuning, picking or looking.
- Write the plan, then run once.
The rule, the line, one run, the result recorded whatever it is. A second look or a second rule makes the stretch in-sample.
- Read the result with its error bar.
A pass is evidence, not proof. A fail does not prove the rule worthless.
A near miss
Which next step keeps the held-out test honest?
"It passed, so it works"
How many dollars either side of the lead does the error bar reach?
Hint: error bar = lead ± 1.96 × σ × √n, and √625 = 25.
Held out afterwards
What is wrong with that held-out test?
Lesson done
Set the most recent stretch of data aside before any tuning or picking. Do the work on the rest, write down the rule and the line, then run the one chosen rule on the held-out stretch once and record the result whatever it is. A held-out test reports: it does not make a rule work, and a pass is evidence with an error bar, not proof. In the lesson's simulation, across 1,000 markets, the pick's lead averaged $0.78 a year on the held-out years and $0.77 on the next two, while the best of 15 rules run on the held-out years averaged $7.71 there and $0.46 after, all before trading costs. On the market shown the pick passed and then trailed just holding.
Key ideas
- Held-out data
- the most recent stretch of data, set aside before any tuning or picking and left unseen until one chosen rule is run on it once
- Held-out test
- one run of the one chosen rule, settings unchanged, on held-out data, against a line fixed beforehand. A second run, a changed setting or a second rule makes that data in-sample
- Line
- the result a rule must clear to pass a test, fixed before the run
- Forward test
- a rule, unchanged, run on new days as they arrive
Formula
error bar on a held-out lead = lead ± 1.96 × σ × √n
The lead is the total over the held-out days; σ is the standard deviation of one day's lead and n the number of held-out days. It applies to one rule run once, as planned, when one day's lead does not change the next day's. Around the best of several runs it sits on a flattered number.