Backtesting Essentials · Lesson 9
In-sample, out-of-sample and walk-forward
How to split the past so that a tuned rule meets data its tuning never saw.
Know the basics? Read the lesson notesCut the past before you tune
A backtest runs a trading rule over past prices to see what it would have made. Here the prices are simulated, made up by a computer: eight years, 2015 to 2022.
Each day, a rule holds the market or stays out. Holding a day puts $100 in at the day's first price and takes it out at its last. A rule's result is its days' gains and losses added up, with no trading costs. Just holding every day is the yardstick: a rule's lead is its result minus just holding's over the same days.
Each evening the market publishes a made-up number, the reading. We built this market, so we know what is real in it: the next day's move tends to go a little the reading's way. The month changes nothing.
The lesson's rule sorts days by two conditions: was yesterday's reading above 0, and which month is it? That makes 2 × 12 = 24 kinds of day, each with a setting: hold, or stay out. Tuning finds the settings with the best backtest: hold a kind of day if its days added up to a gain.
A tuned rule looks good on the years it was tuned on, whether its settings fit something real or luck. So cut the years in two before tuning. The earlier part is in-sample: the years the rule is tuned on. The later part is kept aside.
Eight simulated years, cut in two
Dark: in-sample. Hatched: kept aside.
In-sample the tuned rule made $113 and just holding $53: a lead of 113 − 53 = $60. The lesson keeps this cut from here on.
Four years the tuning never saw
The years kept aside are out-of-sample: they played no part in tuning the rule. Lesson B8 called the two parts tuning data and new data. The rule keeps its 24 settings.
The tuned rule in-sample, then out-of-sample (simulated)
Colour: the tuned rule. Grey: just holding. Both restart from $0 at the cut.
Out-of-sample the rule made $1. Just holding made $0 there, against $53 in-sample, so the raw results cannot be compared. The lead takes the market out: 1 − 0 = $1, against $60 in-sample.
The reading is real, but its effect is small next to four years of luck, and each of the 24 settings was tuned on about 43 days (1,039 ÷ 24), so the tuning mostly fit chance.
On 1,000 more markets built the same way, the lead averaged $53 in-sample and −$1 out-of-sample. It was above zero out-of-sample in 483 of them: close to a coin flip.
For the precise
To one decimal, in-sample then out-of-sample: the rule $113.5 and $1.4, just holding $53.4 and −$0.3. Whole-dollar figures round halves up, and every lead is taken between the whole-dollar figures shown. The in-sample years hold 1,039 trading days, about 43 for each setting, and the out-of-sample years 1,044; tuning set 13 of the 24 kinds of day to hold. Over the 1,000 markets the averages were $84 and $29 for the rule and $31 and $30 for just holding, so the average leads are 84 − 31 = $53 and 29 − 30 = −$1. The out-of-sample lead was below the in-sample lead in 975 markets. A rule with the reading alone (2 settings) averaged $51 in-sample and $43 out-of-sample, and beat just holding out-of-sample in 608 markets.Why cut by time?
Why not mix the days instead: day one in-sample, day two out-of-sample, and so on?
Try both splits on a rule that means nothing: the month alone, 12 settings. One change for this screen: a trade lasts five days. $100 goes in at a day's first price and comes out at the last price four trading days later. A new trade starts on every day of a held month; just holding starts one every day. A lead here adds up every trade, with up to five open at once, so the dollars run larger.
Which split will show the bigger out-of-sample lead?
The month rule, split two ways (simulated)
Top: six five-day trades, split every other day. Dark: in-sample. Colour: out-of-sample.
You guessed the cut by time. It was the other one.
You guessed every other day, and it was.
You guessed the same. They came out far apart.
A rule tuned on luck looks good on the trades it was tuned on. Cut by time, the lead fell from $147 in-sample to $24 out-of-sample. Split every other day, it went from $84 to $112: that test says a rule built on nothing held up.
Why: an out-of-sample trade shares four of its five days with the in-sample trade that started the day before, so its result is mostly a copy, luck included. The in-sample trades give part of the out-of-sample answer away: the split leaks. Cut by time it does not: this test starts no trade in the four days before the cut, so no trade crosses it.
A mixed split leaks whenever neighbouring days share something. A cut by time also copies how a rule is used: tuned on the past, traded after it.
For the precise
On the 1,000 other markets the out-of-sample lead was above zero in 940 for the every-other-day split and in 270 for the cut by time. The 270 is below half because the made-up market rises over time and a rule built on nothing sits out part of it (it held 7.5 of the 12 months on average), so it tends to trail just holding. The 24-setting rule of the screen before sat out days too, but had the reading to make up for it. Results behind the leads, in-sample then out-of-sample. Cut by time: the rule $385 and $56, just holding $238 and $32. Every other day: the rule $247 and $220, just holding $163 and $108. The numbers are larger than elsewhere in the lesson because up to five trades are open at once. No trade starts on the last four days of a stretch, so none runs past its end. Averages over the 1,000 markets: cut by time, a lead of $145 in-sample and −$63 out-of-sample; every other day, $90 and $80. Cut by time, the rule's out-of-sample result averaged $118 against just holding's $181: about the share of the months it held. With one-day trades, where neighbouring trades share no day, the every-other-day split leaked nothing in this simulation: the month rule's lead went from $34 to $5 by time and from $39 to $2 every other day.Fifteen rules, one pick
Back to one-day holds, cut by time. After a lead of $1, the tempting move is to try other rules. Here are 15: every mix of four conditions except none (2 × 2 × 2 × 2 − 1 = 15). The conditions are the reading, the month, the weekday, and whether yesterday's price rose. Only the reading is real.
Each is tuned on the in-sample years alone, all 15 run on the out-of-sample years, and you keep the best.
15 tuned rules, then the pick (simulated)
Each dot: one rule's four-year result on 2019–2022, in the order run. Dashed: just holding. Colour: the best.
The pick was the rule with the reading alone, the one real condition: $26 on 2019–2022, where just holding made $0. Over four years that lead is 26 ÷ 4 = $6.50 a year.
No rule was tuned on those years, but the pick was made on them: picking one rule from several is a second way a stretch gets used. So $26 is the best of 15 on the years that picked it, and the best of many tries is partly luck, even when the rule is real (lesson B7, The best of many). On the 1,000 other markets the pick was this rule in only 390. For the pick, 2019–2022 are now in-sample.
The simulated market runs on for two more years, 2023 and 2024, that nothing has touched. There the pick made $30 and just holding $41: a lead of 30 − 41 = −$11, or −11 ÷ 2 = −$5.50 a year. The reading's effect is small, and two years are short.
On the 1,000 other markets the pick's lead averaged $5.7 a year on the years that picked it and $1.2 a year on the two after. It was lower after in 668 of them.
For the precise
It was higher after in 265 markets and the same in 67, where the pick's tuned settings held every day, as just holding does, so its lead was zero both times. For a rule fixed before any out-of-sample run (the lesson's 24-setting rule) the lead was lower after in 497 markets and higher in 503, averaging −$0.4 and −$0.3 a year: with no picking, no fall on average. The pick used the reading, alone or with other conditions, in 811 markets, so picking this way often found the real condition; its lead on the years that picked it was still too high on average. On the lesson's market 6 of the 15 rules beat just holding on 2019–2022; the worst made −$39. The rule with the best in-sample result ($256, with 240 settings) made −$22 there. The lesson's own rule made $5 on 2023–2024. Those two years hold 522 trading days.Walk-forward: cut, test, move on
One cut gives one out-of-sample stretch, and a few years are one draw of luck.
Walk-forward repeats the cut. Tune the rule on a tuning window, here three years. Run it on the test stretch right after, here one year. Then move both one year on and repeat. Each test stretch is out-of-sample for the settings that trade it.
The rule is the 24-setting one, on a second simulated market built the same way: ten years, 2015 to 2024, that nothing above has touched.
Walk-forward on ten simulated years
Dark: tuning window. Colour: test stretch. Below: the test stretches so far, joined; grey is just holding.
Joined end to end, the seven test stretches (2018 to 2024) make one out-of-sample record, the walk-forward record: $51, where just holding made $43 over the same years. On this market, a lead of 51 − 43 = $8.
The record against a plain backtest
A plain backtest tunes once, on all the data. For comparison, do that with the same rule and market, and read the same seven years.
One rule on 2018–2024, three ways (simulated)
Tuned on all ten years, the rule shows $107 on 2018–2024: a lead of 107 − 43 = $64. Those years are in-sample for it.
On the 1,000 other markets the plain backtest's lead on those years averaged $60 (114 − 54) and the record's −$8 (46 − 54). It showed the more in all 1,000.
So on this market the record led just holding by $8, and across the 1,000 it trailed by $8 on average: it led in 383 and trailed in 617. One record has luck in it, the rule's real effect is small, and its 24 settings mostly fit chance.
Walk-forward does not make a rule work. It shows what a tuned rule did on days its tuning could not see.
For the precise
The record to one decimal is $50.6. The first tuning window holds 778 trading days, about 32 for each setting. The averages behind the leads: the plain backtest $114, the record $46, just holding $54. A rule with the reading alone (2 settings), walked forward the same way, led just holding in 693 markets, with an average record of $72; on the second market its record was $74, a lead of $31.In-sample, out-of-sample and the lead a year
In-sample data is the stretch used to tune a rule or to pick it from several. Out-of-sample data is a later stretch that played no part in either. An out-of-sample test runs the rule there, settings unchanged.
Taking off just holding removes what the market did. Dividing by the years lets stretches of different length be compared. This course calls the result the lead a year.
- rule
- a rule's result on a stretch of days
- hold
- just holding's result on the same days
- years
- the length of the stretch, in years
- lead a year
- the rule's lead over just holding for each year of the stretch (in the lesson: dollars on $100 held a day, before costs); below zero means behind
On the four years that picked it: (26 − 0) ÷ 4 = $6.50 a year. On the two years after: (30 − 41) ÷ 2 = −$5.50 a year.
- the out-of-sample stretch comes after the in-sample one and played no part in tuning the rule or in picking it.
The out-of-sample result has been used to re-tune or to pick between rules. That stretch is then in-sample, and a last check needs data nothing has touched (lesson B10, The held-out test).
Over five out-of-sample years a rule made $70 and just holding made $40. What is the rule's lead a year?
Counting a walk-forward
Walk-forward analysis: tune a rule on a window of past data, test it on the stretch right after, move both forward by the length of the test stretch, and repeat to the end of the data.
- N
- the length of all the data
- T
- the length of the tuning window, which stays the same as the window moves
- S
- the length of one test stretch, which is also how far the window moves at each step. N, T and S are in the same unit, such as years
- rounded down
- a last stretch shorter than S is not used, so the record is test stretches × S long
Ten years of data, a three-year tuning window, one-year test stretches: (10 − 3) ÷ 1 = 7 test stretches, and a record of 7 × 1 = 7 years.
- the window moves by exactly S, so test stretches neither overlap nor leave gaps.
The record stops being out-of-sample when a trade starts in the tuning window and ends in the test stretch (the leak of the five-day test), or when the record is used to pick a rule or a window length.
You have 12 years of data, a tuning window of 4 years and test stretches of 2 years. How many test stretches?
What this means for your backtest
- Cut by time, before you tune.
Tune and pick on the earlier years only. When trades or inputs span several days, never mix the days: neighbours share what they cover.
- Read an out-of-sample result once.
Re-tune after it, pick between rules by it, or drop the losers by it, and that stretch is in-sample.
- Ask what a walk-forward covered.
The window lengths, the number of test stretches, and how many rules were tried this way.
"Tested out-of-sample"
What is the 2021–2024 result of the rule shown?
Plan your own walk-forward
How many test stretches will the record have?
Hint: test stretches = (N − T) ÷ S, rounded down.
A split at random
What is wrong with the test?
Lesson done
Cut the past by time before tuning: tune a rule, or pick one of several, on the earlier, in-sample part, then run it unchanged on the later, out-of-sample part. A stretch stays out-of-sample only until its result is used to tune or to pick. Walk-forward repeats the cut and joins the test stretches into one out-of-sample record: an honest record of a tuned rule, not a way to make one work. In the lesson's simulation it led just holding on the market shown and trailed it on average across 1,000.
Key ideas
- In-sample
- the stretch of past data used to tune a rule or to pick it from several (lesson B8's tuning data)
- Out-of-sample
- a later stretch of data that played no part in tuning a rule or in picking it (lesson B8's new data). Once its result is used to tune or to pick, it is in-sample
- Lead
- a rule's result minus just holding's result over the same days
- Walk-forward
- tuning a rule on a window of past data (the tuning window), testing it on the stretch right after (the test stretch), then moving both forward and repeating
- Walk-forward record
- a walk-forward's test stretches joined in order: out-of-sample as long as nothing was picked by looking at it
Formulas
lead a year = (rule − hold) ÷ years
test stretches = (N − T) ÷ S, rounded down
Lead a year: both results over the same days. Test stretches: N, T and S in one unit, the window moving by S.