Backtesting Essentials · Lesson 16
From backtest to forward test
Days that did not exist when the rule was fixed: what a forward test can show, why a rule picked as the best of several backtests tends to do worse there, and how to decide in advance what would make you stop.
Know the basics? Read the lesson notesTwenty rules, eight years
As in earlier lessons, a backtest runs a trading rule over past prices to see what it would have made. Here the prices are simulated: one made-up market, 2015 to 2024. You stand at the end of 2022, with eight years of prices.
Each day a rule is in the market or out. In: $100 goes in at the day's first price and comes out at its last. No trading costs are taken off, so every result here is before costs. A rule's lead is its result minus the result of just holding (being in every day) over the same days (lesson B9).
Each evening this market publishes a made-up number, the reading, usually between about −2 and +2. We built it so the next day's move tends to go a little its way: each unit of the reading adds 0.10% to that move, where a typical day moves about 1.03% up or down. You also have 19 more made-up readings that push nothing. Rule 1 is in the market the day after the market's reading was above 0; rules 2 to 20 do the same with the 19 others. Backtest all twenty, and keep the best: the pick.
20 rules on 2015–2022 (simulated, before costs)
Dots: each rule's lead a year. Colour: the pick. Dashed: just holding.
The pick is rule 1, the one that reads the market's reading. It made $58 where just holding lost $61: a lead of 58 + 61 = $119 over eight years, or 119 ÷ 8 = $14.88 a year, before costs.
It is the best of 20 backtests, and the best of many tries is partly luck (lesson B7).
Days that did not exist
At the end of 2022 the pick is fixed: no setting will change. A forward test runs the rule, unchanged, on new days as they arrive (lesson B10's words).
A held-out test (lesson B10) runs the rule once on past days set aside before the work: those days existed, so someone could have looked at them. Forward days did not exist when the rule was fixed, so nobody could have looked at them, tuned on them or picked with them.
Which of these is a forward test of the pick?
Will it match the backtest?
Now let 2023 and 2024 happen, with the pick unchanged. In its backtest it led by $14.88 a year. Guess first: over the two forward years, what will its lead a year be?
The pick's lead, added up day by day (simulated, before costs)
Dark: the backtest. Colour: the forward test. Dashed: the backtest's pace, carried on.
You guessed higher. On this market it was a little lower, about the same: $14 a year against $14.88.
You guessed about the same, and on this market it was: $14 a year against $14.88.
You guessed lower. On this market it was only a little lower, about the same: $14 a year against $14.88.
Over 2023–2024 the pick made $26 and just holding lost $2: a lead of 26 + 2 = $28, or 28 ÷ 2 = $14 a year, before costs.
That is one market. On 1,000 more markets built the same way, the pick's lead averaged $4.71 a year in the backtest and $2.73 in the forward test, before costs.
Why lower: the pick is partly luck
A pick's backtest is the best of several tries, so part of its lead is luck that new days don't repeat. To see how much, take the same 1,000 markets twice: with the reading's built-in nudge (call it the push), and with the push removed (the same prices otherwise). Without the push no rule has anything real, yet a rule out of the market about half the days still trails just holding by about $4 a year here, because these markets rise on average, so sitting out about half the days misses about half the rise. The figure opens on "With the push"; switch to see the other.
Average lead a year, 1,000 markets (simulated, before costs)
Dark: the backtest, 2015–2022. Colour: the forward test, 2023–2024.
With the push, the pick's lead fell from $4.71 a year in the backtest to $2.73 forward. Rule 1 on its own, never chosen from the twenty, averaged $4.37 and $4.33: about the same, because nothing chose it for a lucky backtest. The fall comes from the picking: in some markets the pick was a rule with nothing real in it and a lucky backtest.
Without the push no rule has anything real. The pick still averaged $1.53 a year in the backtest, and then trailed just holding by $3.89 a year forward: the sitting-out handicap, not luck.
So when a rule was picked as the best of several on the same backtest, expect its forward lead to be lower than its backtest's on average, though not every time: with the push it was higher in 423 of the 1,000 markets.
For the precise
With the push the pick was rule 1 in 812 of the 1,000 markets (a blind draw from 20 would get it in about 50), and a rule with nothing real in it in the other 188.Decide when to stop, before you look
A forward test needs a plan written before its first day: how long it runs, and what result would end it. That is a stop rule, for example: after 65 trading days (days the market is open: 65 is about three months, and a year has about 260), stop the rule if its lead is below a line. A line drawn after the result can be drawn to fit it (lesson B10).
To count what a stop rule does, take rule 1, whose truth we know: with the push it has a real lead, and on the same markets without the push it has nothing real. Choose your line, then run three months forward on all 1,000 markets.
Rule 1 after 65 forward days, 1,000 markets (simulated, before costs)
With your line at −$5, the stop rule stopped rule 1 with its real push in 137 of the 1,000 markets, and let it run on without the push in 768.
With your line at $0, the stop rule stopped rule 1 with its real push in 432 of the 1,000 markets, and let it run on without the push in 395.
With your line at +$5, the stop rule stopped rule 1 with its real push in 793 of the 1,000 markets, and let it run on without the push in 115.
Letting the empty rule run on means its lead reached the line by luck, despite trailing just holding by about $4 a year on average. A lower line stops a real lead less often and lets an empty rule run on more often (−$5: 137 stopped, 768 ran on; +$5: 793 and 115), and after three months no line does both well; the next screen shows why.
How wide luck is in a forward test
Across the 1,000 markets, rule 1 with the push led by 1.7 cents a day on average, and one day's lead had a standard deviation of $0.73: how far a day's lead typically strays from its average. It is smaller than a day's typical move because the rule is in the market only some days, and its lead is $0 on those. Call that σ (sigma), the usual size of a day's swing. The error bar is the span around the average lead where the result lands about 95 times in 100 (lesson B10); over n days it reaches 1.96 × σ × √n either side (it grows with √n because up and down days partly cancel). Slide the forward test longer.
Where rule 1's forward lead lands 95 times in 100, by the formula (simulated, before costs)
Colour: with the push, centred on 1.7 cents a day. Grey: without it, centred on −1.5 cents a day. Line: the average lead. Band: ± the error bar.
Over four weeks, 20 trading days, the bar reaches 1.96 × 0.73 × √20 = 1.96 × 0.73 × 4.47 = 1.96 × 3.26 = $6.39 either side, around an average lead of 20 × 0.017 = $0.34. A few weeks cannot tell this lead from none.
After two years, 522 days, the bar is $32.69 either side of $8.87, and a line at $0 still stopped the real rule in 299 of 1,000 markets and let the empty one run on in 329. The lead grows with the days; the bar grows only with their square root, so the lead outgrows it, slowly.
For the precise
The bands use the average day and the standard deviation measured over all 2.6 million days, unrounded. Rule 1's actual forward leads with the push fell inside the band in 941, 940, 924, 932 and 948 of 1,000 markets at 20, 65, 130, 260 and 522 days, where the formula says about 950: these markets have calm and wild spells, so days are not all alike. Without the push rule 1 led by −1.5 cents a day on average.How many days a forward test needs
Over n days a rule whose true average lead is d a day leads by d × n on average, and the error bar reaches 1.96 × σ × √n. Set the two equal: d × n = 1.96 × σ × √n. Divide both sides by d × √n: √n = 1.96 × σ ÷ d. Square both sides, and you have a sample-size calculation: how many days a test needs before an expected lead reaches the edge of its error bar. At that length, the average lead equals the bar, so the bar's lower edge just touches $0.
- n
- the number of forward days at which a lead equal to its average, d × n, reaches the edge of its 95% error bar
- d
- the rule's true average lead a day. It is unknown: a picked backtest's figure is flattered, so n from it comes out too small
- σ
- the standard deviation of one day's lead
- 1.96
- the multiplier for a 95% error bar (lessons B5 and B10)
σ = $0.73 and d = $0.017. 1.96 × 0.73 = 1.4308. 1.4308 ÷ 0.017 = 84.16. 84.16² = 7,082.9: about 7,083 days, and at about 260 trading days a year, 7,083 ÷ 260 = 27.2 years.
Even then, a forward test lands above its average about half the time and below it half the time, so it clears zero by its whole bar only about half the time. The lead enters squared: twice the lead needs a quarter of the days.
- one rule, fixed, run once as planned;
- one day's lead does not change the next day's;
- d and σ stay the same on the forward days.
d is taken from a picked backtest, which is flattered, so n comes out too small. It also breaks when the market changes under the rule (lesson C3, Regimes and decay).
New numbers. A rule's true average lead is $0.05 a day, and one day's lead has σ = $0.60. About how many forward days does it need?
What a forward test can't do
A forward test is one draw, with its error bar. It can't promise the market stays the same: here the push never changes, while in a real market an effect can fade (lesson C3). And it can become the best of many again. Try it: forward test all twenty rules on 2023 and keep the best.
20 rules forward tested on 2023 (simulated, before costs)
Dots: each rule's lead in 2023 on the market shown. Colour: the best of 20. Bars: the best of 20's average lead across the 1,000 markets with the push.
On this market the best of 20 in 2023 was rule 14, one with nothing real in it: a lead of $14 in 2023, then $5 in 2024.
On the 1,000 markets with the push, the best of 20 forward tests led by $12.19 on average in 2023, and trailed by $1.54 in 2024, in this simulation, before costs. Part of the fall below zero is the sitting-out handicap: a rule with nothing real trails holding by about $4 a year. Twenty forward tests with the best kept is lesson B7's best of many: the forward days picked the rule, so its result there is partly luck.
For the precise
The best of 20 in 2023 led by less in 2024 in 827 of the 1,000 markets. It was rule 1 in 249; a blind draw: about 50.What this means for your backtest
- Write the stop rule before the first forward day.
How long the test runs, the line, and what happens below it. A line drawn after the result can be drawn to fit it.
- Expect a rule picked as the best of several on the same backtest to do worse forward, on average.
Its backtest is partly luck. In this lesson's simulation, with the push, the pick's lead fell from $4.71 to $2.73 a year on average, before costs.
- Read a forward test with its error bar, and count every one you run.
Weeks can't tell a small lead from none: n = (1.96 × σ ÷ d)². Many forward tests with the best kept are the best of many again.
"Forward test it for a month"
Taking the claim at its word, about how many forward days would the rule need for a lead at its average to reach the edge of its error bar?
Hint: n = (1.96 × σ ÷ d)², with σ = 1.00 and d = 0.10.
Three weeks in
Which next step keeps the forward test honest?
The best live result
What is the first problem to raise with that $15?
Lesson done
A forward test runs a rule, unchanged, on days that did not exist when it was fixed. When the rule was picked as the best of several on the same backtest, expect its forward result to be lower on average, though not every time: in the lesson's simulation the pick's lead averaged $4.71 a year in the backtest and $2.73 forward, before costs. Write the stop rule before the first forward day, and read the result with its error bar: for that push, four weeks could not tell the lead from none. Many forward tests with the best kept are the best of many again.
Key ideas
- Forward test
- a rule, unchanged, run on new days as they arrive: days that did not exist when the rule was fixed, so nothing in it can have come from them
- Stop rule
- a plan written before a forward test's first day: how long the test runs, and the result below which the rule is stopped
- Sample-size calculation
- working out how many days a test needs before an expected lead reaches the edge of its error bar
Formula
days a forward test needs: n = (1.96 × σ ÷ d)²
d is the rule's true average lead a day and σ the standard deviation of one day's lead. At n days the average lead equals the 95% error bar, so the bar's lower edge just touches $0. It holds for one fixed rule run once, with days that don't change each other and a market that stays the same; d from a picked backtest gives too few days.