Backtesting Essentials · Lesson 8
Overfitting
When a rule memorises the past: why it can look far better on the data it was tuned on than on data it has never seen.
Know the basics? Read the lesson notesA rule with two settings
A backtest runs a trading rule over past prices to see what it would have made. This lesson's prices are simulated: a computer made them up. Start with five years, 2015 to 2019.
Each day, a rule either holds the market or stays out. Holding a day means putting $100 in at the day's first price, its open, and taking it out at its last, its close. A rule's result is what its days made and lost, added up, with no trading costs. Just holding every day made $24 over the five years.
We built this market, so we know what is real in it. One thing: after each close the market publishes a made-up number, the reading, and the next day's move tends to go a little the reading's way, up after a reading above 0 and down after one below. The weekday and the month change nothing.
The condition "was yesterday's reading above 0?" sorts the days into two kinds: days after a high reading and days after a low one. The rule gets one setting for each kind: hold, or stay out. Picking the settings that give the best backtest is called tuning. Here that means: hold a kind of day if its days added up to a gain over these five years.
One rule, tuned on 2015–2019 (simulated)
Colour: the rule's result so far. Grey: just holding. Boxes: its settings, filled for hold.
Days after a high reading added up to +$37, so their setting is hold. Days after a low reading added up to −$13, so theirs is stay out. Just holding took both: 37 − 13 = $24.
Keep adding conditions
Can more conditions beat $37? Add the weekday: 2 × 5 = 10 kinds of day, each with its own setting. Then add the month: 10 × 12 = 120 kinds, 120 settings. So each condition added multiplies the settings. Each time, every setting is tuned again on the same five years.
The same rule with more settings, tuned on 2015–2019 (simulated)
Lighter lines: the rules before. Boxes: its settings, filled for hold.
Each condition lifted the result: $37 with the reading, $65 with the weekday, $150 with the month. The month added the most, 150 − 65 = $85, and the month changes nothing in this market (a later screen shows how that can be). The reading, the one real thing, had added 37 − 24 = $13.
A guess before the new years
The 120-setting rule made $150 on 2015 to 2019. Call those the tuning years: the years its settings were picked on.
The market runs on for five more years, 2020 to 2024, built the same way. The tuning never saw them: the new years. The rule keeps its 120 settings as tuned. What will it make there?
Five years it has never seen
Left: the tuning years again. Right, shaded: the new years. Each side starts from $0. Run the new years.
The 120-setting rule: tuning years, then new years (simulated)
Colour: the tuned rule. Grey: just holding.
On the new years the rule made $16. Just holding made $27 there, so the tuned rule did 27 − 16 = $11 worse than no rule at all. Of its $150, 150 − 16 = $134 was not there on the new years.
The settings were the best possible ones for 2015 to 2019. That is the trouble: most of them fit what luck did in those years, and luck doesn't repeat. A rule whose settings fit the luck in its tuning years is overfit.
Fewer settings, then more
You made four rules on the way: just holding (0 settings), then one, two and three conditions. Slide through them and compare the tuning years with the new years.
Four rules on the tuning years and on the new years (simulated)
Dark: the tuning years. Colour: the same rule, same settings, on the new years.
On the tuning years every added condition lifted the result. When each new condition only splits the old kinds of day further, tuning can never lower the result there: the finer rule can always set its extra settings the same way as the old rule.
On the new years the result first rose, from $27 to $55 with the reading, then fell: $36 with the weekday, $16 with the month. The reading is real, so what tuning found in it was still there. The weekday and month settings fit only luck, and on the new years they did harm.
The figure's last line is the tuning gap: the tuning-years result minus the new-years result. For 2 settings it is below zero, −$18, because luck sometimes favours the new years.
How can a condition that means nothing help?
Look inside the 10-setting rule. Each pair of bars is one of its ten kinds of day: its total on the tuning years, and on the new years.
Ten kinds of day: tuning years and new years (simulated)
Dark: 2015–2019, filled if tuning set the kind to hold, hollow if stay out (a total the rule left out). Colour: 2020–2024.
Against the 2-setting rule, tuning changed three settings and left seven. Take Mondays after a high reading. On the tuning years those 139 days lost $7 in total, so tuning set them to stay out. On the new years the same kind of day made $21, and the rule missed it.
The other two, Mondays and Wednesdays after a low reading, had made $9 and $12, so tuning set them to hold: 7 + 9 + 12 = $28, the step from $37 to $65. On the new years they made $2 and $1. The rule gave up $21 to gain $3: the drop from $55 to $36, give or take a dollar of rounding.
The weekday changes nothing in this market. Why did those Mondays lose money on the tuning years?
Was that just this one market?
One market can be lucky or unlucky. So we built 1,000 more markets the same way, each with its own luck, and did the same on each.
A guess first. In how many of the 1,000 does the 120-setting rule beat the 2-setting rule on the new years?
The four rules on 1,000 simulated markets: average results
Dark: the average result on the tuning years. Colour: the average on the new years.
The 120-setting rule beat the 2-setting rule on the new years in 129 of the 1,000 markets; the 2-setting rule did better in the other 871.
The averages. Tuning years: $36, $62, $77, $177. New years: $38, $56, $44, $28: up with the reading, then down, to 38 − 28 = $10 less than just holding.
For the precise
In all 1,000 markets the 120-setting rule made less on the new years than on its tuning years; it fell below just holding there in 661. The 2-setting rule beat just holding on the new years in 653 markets: five years are short for an effect this small. Its tuning gap was above zero in 545 markets and below zero in 449. Best rule on the new years: no condition in 191 markets, 2 settings in 520, 10 in 207, 120 in 82. Averages to one decimal, tuning years: 35.7, 61.9, 77.1, 176.8; new years: 38.4, 56.3, 43.9, 27.6. Differences in the text are taken between the whole-dollar figures shown, so they can be a dollar away from the difference of the one-decimal figures: 38 − 28 = $10 in the text, 38.4 − 27.6 = 10.8 here.The tuning gap
Overfitting is tuning a rule until its settings fit the noise in the tuning data as well as anything real in it (another name: curve fitting). It shows on new data: data that played no part in tuning the rule or in choosing it.
- tuned
- a rule's result on the data its settings were tuned on
- new
- the same rule's result, with the same settings, on new data of the same length from the same market
- tuning gap
- the part of a tuned result that was not there on new data. A large tuning gap is a sign of an overfit rule
120 settings: 150 − 16 = $134. 2 settings: 37 − 55 = −$18.
How big is large? Read the gap next to what just holding made on both stretches: $24 and $27 here, about the same. A gap of $134 is about five times $27.
One market's tuning gap has luck in it. Averaged over the 1,000 markets it was $6 with 2 settings and $149 with 120.
- the new data played no part in tuning the rule or in choosing it;
- the two stretches are the same length.
The new data has been used to choose a rule: then it is tuning data too (lessons B9, In-sample, out-of-sample and walk-forward, and B10, The held-out test).
New numbers. A rule tuned on eight years made $84 there. With the same settings it made $21 on the next eight years. What is the tuning gap, in dollars?
Every setting multiplies the versions
A version is one full set of settings, and every combination of settings is its own version. To count them, multiply together the number of values each setting can take. Tuning keeps the best version: more versions, more luck fitted (lesson B7, The best of many), and a bigger tuning gap to expect.
- k
- the number of settings a rule has
- c1, c2, …, ck
- how many values the first setting can take, the second, and so on to the last (2 for an on/off setting)
2 settings, each hold or stay out: 2 × 2 = 4 versions. On the tuning years: hold both kinds of day, $24 (just holding); hold only after a high reading, $37; hold only after a low reading, −$13; hold neither, $0. Tuning kept the best, $37.
10 settings: 2 multiplied by itself 10 times, 1,024 versions. 120 settings: 2 multiplied by itself 120 times, far too many to run one by one.
Tuning did not need to: each kind of day adds its own total to the result, so the best setting for each kind gives the best version of all.
- every value a setting could take is counted, the ones never looked at too.
Many versions are near copies: two that differ in one setting out of 120 hold almost the same days, so they add less luck than two separate tries. The count is then an upper limit on the size of the search (lesson B7's caveat; lesson B14, Robustness).
New numbers. A rule has 6 on/off settings. How many versions can tuning choose from?
What this means for your backtest
Three habits follow.
- Count the settings, and the versions they make.
Every setting you can adjust multiplies the versions, and each version is one more try at fitting luck (lesson B7). Count the days or trades behind each setting too: 120 settings on 1,300 days left about 11 each.
- Don't read a result on tuning data as evidence.
On its tuning data a tuned rule looks good whether its settings fit something real or noise: in the lesson's market the month, which changes nothing, added more there than the real reading.
- Judge a tuned rule on data the tuning never saw, and use that data once.
A test like that is a held-out test (lesson B7). We ran four rules on the new years only to show what happens; choosing a rule by them would use them up (lessons B9 and B10).
"Optimised on ten years of data"
What is the first thing to ask?
Count your own search
If you tune all four, how many versions is the tuning choosing from?
Hint: multiply together the number of values each setting can take. On or off is 2 values.
Two rules, one pick
Which statement is right?
Lesson done
Tuning picks the settings that did best on past data, so a tuned rule's result there never falls as settings are added, and usually rises, real or not. In the lesson's simulation, on years the tuning never saw, only the one real condition helped. The first result minus the second is the tuning gap; a large one is a sign of overfitting (one gap has luck in it). So judge a tuned rule on data that played no part in tuning or choosing it.
Key ideas
- Setting
- anything in a trading rule that can be adjusted: a number of days, a level, an on/off switch
- Version
- one full set of settings of a rule
- Tuning
- picking the settings that give a rule its best backtest on a chosen stretch of data, the tuning data
- Noise
- the part of a past result that came from luck and has no reason to repeat
- Overfitting
- tuning a rule until its settings fit the noise in the tuning data as well as anything real in it (also: curve fitting)
- New data
- data that played no part in tuning a rule or in choosing it
- Tuning gap
- a tuned rule's result on its tuning data minus its result, with the same settings, on new data of the same length
Formulas
tuning gap = tuned − new
versions = c1 × c2 × … × ck