Backtesting Essentials · Lesson 1
Inside a backtest engine
Bars, signals, orders, fills: how a backtest walks through the past one day at a time, and where each order really fills.
Know the basics? Read the lesson notesA day is a bar
This lesson runs on SimMarket, the course's simulated market: made-up prices, built to move the way market prices do, with no real market data in them. We use its Market 1, which starts at $100.
A backtest usually reads prices one day at a time, and sums each day up as a bar: its open (the day's first price), its high and low, and its close (the day's last price). Press Play to watch day 2 of Market 1 trade, five minutes at a time, from 09:30 to 16:00.
Market 1, day 2, five minutes at a time (simulated)
Left: the day so far, one small bar per five minutes (each five-minute stretch is drawn like a day, only smaller). Right: the day's bar as it grows. A bar is a line from the low to the high, with a mark to the left at the open and a mark to the right at the latest price.
The day's high, 99.40, came in the five minutes from 11:20. At the time, nobody could know it would be the high: the price might still have gone higher. A bar is complete only at its close. At 16:00 all four prices are known: open 99.19, high 99.40, low 98.76, close 99.27.
A rule that reads one number
At each close, SimMarket also publishes one number for the day, which we call the reading. Day 2's reading was 1.23. It is invented for this course, not a real indicator; a later screen shows what it does to prices.
The rule decides at each close, after reading that day's number. A decision to buy or sell is called a signal. Here are days 1 to 15. Tap every day the rule decides to buy. There are three.
Market 1's readings, days 1 to 15 (simulated)
Each tile is one day and the reading published at its close.
Days 2, 4 and 10. Days 5 and 11 also read above 1, but the rule was already holding a trade, so it waited. It decided to sell at the closes of days 3, 6 and 13, the first readings below 0 after each buy.
The rule says buy. At what price?
At day 2's close the rule reads 1.23 and decides to buy. To buy, it sends an order: an instruction to buy, which fills when someone actually trades with it, at some price.
| Day | Open | Close | Reading |
|---|---|---|---|
| Day 2 | 99.19 | 99.27 | 1.23 |
| Day 3 | 99.10 | 99.32 | −0.75 |
What is the first price a real buy order could get?
The loop, one day at a time
A backtest engine is the program that does this for every day of the data. It is a loop: the same three moves, repeated each day. At the open, fill any order sent the night before. At the close, the day's bar is complete and its reading is out. Then decide, reading only the days so far, and send any order for the next open.
Press Step to move the engine through Market 1's first 30 days, one move at a time. The grey curtain covers the days the engine hasn't reached: it can't see them yet.
The engine on Market 1, days 1 to 30 (simulated)
Top: the loop, with the move it is on. Middle: one bar per day; orange ▲ a buy fill, black ▼ a sell fill; dashed: from the close where the rule decided (small circle) to the open where the order filled. Pink: days the rule holds a trade. Bottom bars: each day's reading, orange on a buy decision, black on a sell decision, with lines at 1 and 0.
Press Step to start day 1.
| Trade | Bought | Sold | Result |
|---|---|---|---|
| No trades yet | |||
The first trade: the buy was decided at day 2's close and filled at day 3's open, 99.10. The sell was decided at day 3's close (reading −0.75) and filled at day 4's open, 99.39. With $10,000 in the trade, the result is 10,000 × (99.39 ÷ 99.10 − 1) = +$29.
Play to day 30 if you like: the engine makes the same three moves every day.
What the engine may read
The engine has reached the close of day 12 and the rule is about to decide. Which days' prices and readings may it use?
Fill at the close, or at the next open?
Some backtests skip the wait. They fill each order at the very close the rule just read, as if the rule could read the close and trade at it in the same instant. Call that a same-close fill. The decisions don't change, only the prices. Switch between the two and watch the trade list.
The same rule, two ways to fill, days 1 to 30 (simulated)
Orange ▲ a buy fill, black ▼ a sell fill. With next-open fills, a dashed line runs from the close where the rule decided (small circle) to the open where the order filled. Pink: days the rule holds a trade. Bottom bars: each day's reading, orange on a buy decision, black on a sell decision.
| Trade | Bought | Sold | Result |
|---|
The same six trades on the same days. Only the fill prices moved, and every result changed: the first trade is +$5 with same-close fills and +$29 with next-open fills. Over these 30 days the next-open fills came out ahead: $294 against $180.
Ten years
Market 1 runs for 2,607 trading days, about ten years, and the rule makes 333 trades with either fill. Give it an account that starts at $10,000 and adds each trade's result, with $10,000 going into every trade. Which fill will end with more in the account?
Run the whole ten years
The same engine, the same 333 decisions, two fill settings. Press Run.
The account after each trade, Market 1, ten years (simulated)
Grey: same-close fills. Coloured: next-open fills. Dashed: the $10,000 start.
The first 30 days said the opposite. Six trades are too few: over a handful of nights, luck decides which fill comes out ahead. Over 333 trades, something else shows. The next screen finds it.
Where the extra came from
We built SimMarket with one planted effect, a small pattern put in on purpose so the course has something to find: a high reading nudges the next day's price up a little, and a low reading nudges it down. Part of each day's move happens overnight, before the next open, and part of the nudge lands overnight too, by design.
So the nights after the rule's decisions lean its way. A same-close fill buys before the night that follows a high reading, and sells before the night that follows a low one. An order sent after the close catches neither night: it fills at the next open, after the night has happened. The daytime part of the nudge is collected by both fills.
To check that the nudge is the reason, switch it off. SimMarket can rebuild Market 1 with exactly the same ups and downs of luck and only the nudge removed. The rule makes the same decisions, because the readings are the same.
The account after each trade, with and without the nudge (simulated)
Grey: same-close fills. Coloured: next-open fills. Dashed: the $10,000 start.
Without the nudge, the difference shrinks from $2,209 to $620, and what's left is about what luck alone can give here. On SimMarket's other ordinary markets with no nudge, the gap came out at about $1,800 on two of them, one each way. Both lines also end lower, as the rule then only collects SimMarket's slow upward drift.
So a same-close fill flatters a rule when what the rule reads really does move the price: part of the move happens before any order can trade at a close you have only just read.
The backtest loop
What you stepped through is called an event-driven backtest: the engine replays the data one bar at a time, in order, and on each bar the rule may use only what was known by then. Using anything from later is called look-ahead (lesson B2). The loop, in plain steps:
for each bar t = 1, 2, … up to the last bar:
open: fill the order sent on bar t − 1 at Ot
close: bar t is complete
decide: the rule reads bars 1 … t, nothing later
if it trades, send an order for Ot+1
- t
- the bar on which the rule decides; its close is the last price the rule has read
- Ot+1
- the open of the next bar: the first price after the decision (Ot is bar t's open, Ct its close)
- Pfill
- the price the order fills at: a next-bar fill. A same-close fill would use Ct, the close the rule just read
The buy was decided on bar t = 2, so it filled at O3 = 99.10. The sell was decided on bar 3, so it filled at O4 = 99.39. Result: 10,000 × (99.39 ÷ 99.10 − 1) = +$29.
- the order is small enough to fill at the open without moving the price;
- the open is a price you could trade at (SimMarket's prices sit halfway between the price to buy and the price to sell, so a real buy pays a little more: lesson A12, Markets and prices);
- trading costs are added on top (lesson A16, Costs).
The rule reads something dated after t, such as a number only published later (lesson B2, Look-ahead bias), or the data itself is wrong on some bar (lesson B4, Data quality).
Try it on new numbers. A rule reads Thursday's bar and decides to buy. Thursday opened at $40.00 and closed at $41.20. Friday opened at $41.50 and closed at $40.90. At what price does a next-bar fill buy?
What a same-close fill assumes
A same-close fill assumes the rule read the close and traded at it in the same instant. The difference between the price when you decide and the price you really get is called implementation shortfall; for one trade here it is rsame − rnext. In this lesson, with no costs, all of it comes from the overnight gap, g = Ot+1 ÷ Ct − 1: the move from the close the rule read to the next open; in real trading, costs and the bid-ask spread add to it. One trade has two gaps, one after each decision. Since Obuy = Cbuy × (1 + gbuy) and Osell = Csell × (1 + gsell), the same-close return is about the next-open return plus gbuy − gsell, and that difference is the shortfall:
- rsame
- the trade's result in %, with same-close fills
- rnext
- the same trade's result in %, with next-open fills
- gbuy
- the overnight gap after the buy decision, in %
- gsell
- the overnight gap after the sell decision, in %
- ≈
- roughly equal: percentage moves multiply rather than add, but for moves of a few percent the difference is tiny
The same-close version comes out ahead when the night after a buy tends to rise (gbuy above 0) and the night after a sell tends to fall (gsell below 0). That happens when what the rule reads moves the price before any order can trade at a close you have only just read. When the reading tells nothing, the gaps are luck and average out.
gbuy = 99.10 ÷ 99.27 − 1 = −0.17%, gsell = 99.39 ÷ 99.32 − 1 = +0.07%, and rnext = 99.39 ÷ 99.10 − 1 = 0.29%. So rsame ≈ 0.29 + (−0.17) − (+0.07) = 0.05%, the +$5 that same-close fills reported. On this trade the nights went against the same-close version.
Over all 333 trades they leaned its way: gbuy averaged +0.049% and gsell −0.018%, so each trade gained about gbuy − gsell = 0.049 − (−0.018) = 0.067%, which is $6.70 on $10,000. Times 333 trades, that is about $2,230, close to the $2,209 you saw.
- each order is a market order (one that takes whatever price is there, lesson A13) and fills at the next open, with no further delay;
- the moves are a few percent at most (the ≈).
Orders take longer than a bar to fill, or are big enough to move the price themselves (lesson D15, Liquidity and market impact).
Try it on new numbers. Same-close fills report a trade at +3.0%. The night after the buy decision the price rose 0.8%; the night after the sell decision it fell 0.5%. About what would next-open fills report, in %?
What this means for your backtest
Every backtest you run or read has a loop like this inside. Three questions follow from it. Turn them over one at a time.
- Where does each order fill?
Find the line that sets the fill price. If it fills at the close the rule just read, the backtest assumes you traded the instant the bar ended. Run it again with next-bar fills and compare.
- Does each decision read only bars 1 to t?
Anything from bar t + 1 on (the next open, tomorrow's close, a number published later) is the future. A rule that reads it can look brilliant and be impossible to trade (lesson B2).
- Can you see the code?
Both answers sit in two lines of a loop like the one on the theory card: the line that reads the data and the line that fills. If you can't read the code behind a result, you can't check either one.
A website's backtest
What is the most useful thing to ask?
Two fills, one trade
What is the trade's result with next-bar fills? Type it in dollars.
Hint: next-bar fills buy at the open after the buy decision and sell at the open after the sell decision. Then the result is 10,000 × (sell price ÷ buy price − 1).
Check the loop
reading[t] means day t's reading, and open[t] means day t's open.)
for t in range(len(days) - 1):
if reading[t] > 1 and not holding:
buy(open[t])What is wrong with it?
open[t] was that morning. Buying there uses the future: a look-ahead. A next-bar fill would be buy(open[t + 1]).close[t] would be a same-close fill: better than open[t], but still a price that is gone by the time the reading is out. Use open[t + 1].Lesson done
A backtest engine is a loop. Each day it fills any waiting order at the open, lets the rule read every day up to the close and nothing later, and sends any new order for the next open. Filling at the close the rule just read assumes you traded in the same instant you decided, and when what the rule reads really moves the price, that makes the result look better than anything you could have traded.
Key ideas
- Bar
- one period's prices: open, high, low and close; complete only at its close
- Signal
- the rule's decision to buy or sell, made at a bar's close
- Order, fill
- the instruction to trade, and the price it actually trades at
- Event-driven backtest
- an engine that replays the data bar by bar; on bar t the rule reads bars 1 to t only
- Look-ahead
- a rule using anything from after bar t (lesson B2)
- Next-bar fill
- the order fills at the next bar's open, the first price after the decision
- Same-close fill
- the order fills at the close the rule just read; optimistic
- Overnight gap
- the move from one close to the next open
- Implementation shortfall
- the difference between the price when you decide and the price you really get
Formulas
Pfill = Ot+1 · g = Ot+1 ÷ Ct − 1 · rsame ≈ rnext + gbuy − gsell