Backtesting Essentials · Lesson 2
Look-ahead bias
Did the test peek at the future? Why a rule that sees what hasn't happened yet looks brilliant, and how to catch it.
Know the basics? Read the lesson notesA rule, ten years, one click
Each day of prices is one bar. Its open is the day's first price, at 09:30, and its close the last, at 16:00. A backtest replays a trading rule over past bars to see what it would have made. As in the last lesson, it walks through the bars in order: at each one the rule decides from what is known so far, and the order fills at the next price available, usually the next bar's open (a next-bar fill).
The rule: on an up day, buy at the open and sell at the close. An up day is a day that closes higher than the day before. Each trade puts in $10,000, and there are no trading costs. Run it on ten years of a simulated market: prices made up by a program, not taken from any real market.
The up-day rule on Market 1, ten years of daily bars (simulated)
The account: $10,000 at the start (dotted line), plus every trade's gain or loss.
$10,000 became $94,665, a gain of $84,665. The rule traded 1,312 times and won 1,101 of them, 83.9%, and its line barely dips.
There was nothing to find
For this lesson the program behind Market 1 is set so that nothing before a day says which way that day will go. Each day's move is a fresh random draw (chance alone sets it, like a coin toss, with no link to which way earlier days went), with a slight upward drift (on average, prices creep up a little). In the morning, no rule can know how the day will end.
Yet this rule won 84 trades in 100. Where do you think its gains came from?
Step through one week
Here is one week of the backtest, moment by moment. The grey curtain covers everything that hadn't happened yet. Each candle is one day: the box runs from the open to the close (hollow if the day closed above its open, filled if below), the thin line from the day's high to its low. On Monday morning, say which price the rule needed. Then press Step to move the clock on.
One week of Market 1, with a curtain over the future (simulated)
Dashed line: the previous day's close; a day that closes above it is an up day. ▶: the rule's buy at the open; a day without ▶ is one it skipped, because it "saw" the day would close down. Orange line: the time now (shown above it); the curtain hides everything after it.
Monday, 09:30: the rule says buy. Which price did it need to decide that?
Last Friday's close is one of the two it compares, and it was known. The other is Monday's close, which isn't known until 16:00. That one is behind the curtain: the peek.
The open is known at 09:30; it is the price the order fills at. To know whether Monday is an up day, the rule needs Monday's close, which isn't known until 16:00. That one is behind the curtain: the peek.
Right. An up day closes above the day before, so the rule needs Monday's close, which isn't known until 16:00. It is behind the curtain: the peek.
Every morning, the rule decided with a price from 16:00. To know at 09:30 whether Monday is an up day, you need Monday's close, and nobody has it until the day is over. So the test bought only on days that were going to close up. That week it won all 3 of its trades.
This is look-ahead bias: a backtest that uses information which wasn't available at the moment of the decision. It explains the ten years, and luck can't: by luck alone, a rule like this would win about half its 1,312 trades, not 1,101. (It still lost some: an up day can close below its own open when it opened even higher.)
Fix it, then guess
The as-tested run broke the rule from the last lesson: it let a day's close decide a trade at that same day's open. The fix is to decide at the close, once the day is finished, and buy at the next open. The fixed rule: after an up day, buy at the next day's open and sell at that next day's close. Everything it reads is already known when it buys.
Over the same ten years, what will the fixed rule make?
As tested, then fixed
The same market, the same ten years, the same $10,000 a trade. The only change is one day of waiting. Switch between the two.
The up-day rule as tested and fixed, Market 1 (simulated)
Coloured: the version you picked. Grey: the other one. Dotted line: the $10,000 start.
That small gain is no edge either (an edge is a real reason to make money on average). The market was built to drift up a little: on the average day it rose 0.018% from open to close. A rule that holds on 1,311 average days would make about 1,311 × 0.018% × $10,000 ≈ $2,400, and luck explains the rest. Nearly all of the $84,665 came from the peek.
Find the peek in three rules
That peek was plain once you stepped through it. Most are better hidden. Each rule below fills at the next price, so the leak must hide in the data. Bars can be any length: a 5-minute bar holds five minutes of prices. In a spreadsheet, each day is one row. An indicator is a number worked out from prices, such as an average; a 20-day average is the average of the last 20 closes. In each rule, tap the part that uses something not yet known at the moment of the decision.
Not this part. When the report comes out is a date on the calendar, known in advance.
Not this part. Buying is the action; the question is what the rule read to decide it.
The peek. Reports like this are often corrected in the weeks after they first come out, and many data files keep only the corrected number. On the morning itself, people saw the first number, which could be different. This leak is called revised data; the fix is the number as first published.
Not this part. Last month's report came out a month earlier, so it was known (as long as it is the number published then, not a later correction).
Not this part. Buying at 10:00 is the action; the question is what the rule read to decide it.
The peek. At 10:00 the day still has six hours to go. Today's daily candle is finished at 16:00, and only then do you know its close and so whether it is hollow. This is an unfinished higher-timeframe bar: a longer bar used before it is complete. The fix is yesterday's daily candle, or today's price so far.
Not this part. The 5-minute bar that started at 09:55 finished at 10:00, so its rise is known.
Not this part. Buying at the open is the action; the question is what the rule read to decide it.
Not this part. Yesterday's close was known since 16:00 yesterday.
The peek. The average in today's row includes today's close, which comes at 16:00. The rule reads it at the open. This is an indicator off by one row; the fix is the average from yesterday's row. A peek mixed in with older numbers is smaller: today's close is one of 20 averaged, so it moves the average only a little. A small peek still flatters the result, and looks more believable.
Each is a common leak: revised data, a longer bar that isn't finished, and an indicator off by one row. With using the same bar's close (the up-day rule), these are the four to check first.
Scramble the future
You can't always read a rule's code. Here is a test that finds a peek without reading it. For each decision, scramble what came after it: swap that day's move from open to close for the move of another day chosen at random (one after the decision). Any such day will do: the rule couldn't have known its move at the time either, so an honest rule can't care which one it gets. The fixed rule's decision for a day reads only closes from before that day, so scrambling that day can't touch it. Then ask the rule again. A rule that reads only the past gives the same answer, because its past hasn't changed. A rule that peeks may change its mind.
40 days of decisions, before and after scrambling the future (simulated)
Filled square: buy. Empty square: don't buy. Squares show each day's original decision; a mark underneath means scrambling the future changed it. Both rows use the same 40 days; the fixed row is the as-tested row shifted one day later.
Use only what was known: shift by one bar
At every moment, a backtest may use only what was known at that moment. Everything known up to a time t is called the information set at t, and a rule that reads anything outside it has look-ahead bias. For a rule that fills at a bar's open, the rule's answer must come from bars up to the one before. So the answer is shifted by one bar before it is traded:
- t
- the bar you trade; the order fills at its open
- signalt−1
- the rule's answer (1 = buy, 0 = don't), worked out at the close of bar t − 1 from bars up to t − 1 only
- positiont
- whether you hold during bar t (1 = hold, 0 = don't)
Its signal at a close is 1 when that close is above the one before. As tested, the backtest used positiont = signalt: Monday's position came from Monday's close, which arrives after Monday's 09:30 fill. Fixed, positiont = signalt−1: Monday's position comes from Friday's close, known since Friday at 16:00. One bar of shift took the ten years from $84,665 to $3,372.
In numbers: closes on days 1 to 5 of 10, 11, 10, 12, 13 give signals for days 2 to 5 of 1, 0, 1, 1 (11 is above 10, 10 isn't above 11, and so on). Shifted by one bar, those become the positions for days 3 to 6: 1, 0, 1, 1.
- every input carries the time it really became known: a report on the day it was published, not the end of the month it describes;
- the fill comes after the decision (here, the next open).
The data's own times are wrong: for example, a 5-minute bar from 10:00 to 10:05 filed under 10:00, though its close is known only at 10:05 (lesson B4, Data quality). The future can also leak in through which markets you test (lesson B3, Survivorship bias).
Try it on new numbers. Closes on days 1 to 5 are 20, 21, 22, 21, 23. The fixed up-day rule takes its positions for days 3 to 6 from the signals of days 2 to 5. On which of days 3 to 6 does it not hold?
Two tests for a leak
A rule's decision for bar t, made at its open, must not change when anything after that moment changes. Testing exactly that is the strongest check for look-ahead, a leakage test. You ran it by scrambling:
- positiont
- the rule's decision for bar t, made at its open (1 = buy, 0 = don't): the decisions you scrambled
- past
- everything known at bar t's open
- future, future′
- what really happened after that moment, and a scrambled version of it
If the two sides differ at any t, the rule read the future.
As tested, the rule says buy if today's close is above yesterday's. Yesterday closed at 10 and today at 11, so its position for today is 1. Scramble today's move so that today closes at 9.5 instead, and the position becomes 0. The two sides differ: that is a peek. The fixed rule's position for today compares yesterday's close with the day before's; the scramble doesn't touch either, so its two sides always match.
A quicker warning sign: re-run the rule with every input one more bar late. A rule that peeks usually collapses: the as-tested rule, one bar late, is the fixed rule, and $84,665 fell to $3,372. A collapse alone isn't proof, because a real effect that lasts only one bar would fade too. The scramble test is the one that settles it for leaks in the data.
- you scramble every input the rule reads, including other files (reports, other markets, longer bars);
- the rule gives the same answer every time it sees the same inputs (no random numbers of its own).
The peek isn't in the data: settings or markets chosen after seeing the results. Scrambling can't catch that (lesson B7, The best of many; lesson B10, The held-out test).
You check a friend's rule. Scrambling the future changes 0 of its 900 signals. With every input one bar later, its gain falls from $30,000 to $26,000. What do you conclude?
What this means for your backtest
Four habits keep the future out of a test. Turn them over one at a time.
- Write down when each input becomes known.
Next to every number a rule reads, note when it exists: a close at 16:00, a report on the day it is first published (with the number published that day), a daily candle at the end of the day. The fill must come after all of them.
- Shift by one bar unless you can say why not.
In code, the line that turns a signal into a position is where most leaks live. Position at bar t = signal at bar t − 1.
- Scramble the future once for every new rule.
If any past decision changes, the rule reads the future, wherever the bug is hidden.
- Treat a result that looks too good as a bug report.
When a simple rule wins most of its trades with a line that barely dips, check the timing before anything else.
A seller's backtest
What do you make of it?
A monthly report
For the trade on 1 June, which month's report is the latest one the rule may use? Type the month's number (January = 1).
Hint: a report becomes known when it is published, not when its month ends. Which reports were out by 1 June?
"It's only one day in 50"
Compared with the fixed version on the same data, what do you expect the as-tested backtest to show?
A 20-day average in your spreadsheet
With the shift done right, which 20 closes go into that average? Type the first day of the 20.
Hint: which is the last close known at day 101's open?
Lesson done
A backtest may use only what was known at each moment. A rule that reads a price, a report or a bar before it exists looks brilliant and is worthless: on a market built with nothing to find, one bar of peeking turned $10,000 into $94,665. Shift every signal by one bar, note when each input becomes known, and scramble the future to check.
Key ideas
- Look-ahead bias
- a backtest that uses information which wasn't available at the moment of the decision
- Information set
- everything known up to a moment; a rule may read only from it
- Shift by one bar
- trade bar t on the signal from bar t − 1, so the fill comes after everything the rule read
- The four common leaks
- using the same bar's close (the up-day rule); revised data; a longer bar that isn't finished; an indicator off by one row
- Revised data
- numbers corrected after they first came out; a backtest should use them as first published
- Leakage test
- scramble everything after each decision; if any decision changes, the rule read the future
Formulas
positiont = signalt−1 · positiont(past, future) = positiont(past, future′)