Backtesting Essentials · Lesson 4
Data quality
Four errors hiding in a price file, how to find each, and which moves a result most.
Know the basics? Read the lesson notesTen years, as delivered
This lesson runs on SimMarket, the course's simulated market: made-up prices, no real market data. Here its Market 1 stands for the shares of one made-up company, over ten years (2,607 trading days). Each day has an open, its first price, and a close, its last.
Below is Market 1 as a careless data source might deliver it: raw data, the prices exactly as recorded, nothing corrected. A backtest replays a rule on past prices. The simplest rule is to hold: buy $10,000 of shares at the first day's open and keep them. Press Hold.
Holding Market 1 for ten years, raw data (simulated)
The account: what the shares are worth at each day's close. Dashed: the $10,000 start.
The backtest says holding lost money: $10,000 became $7,707, since 10,000 × 188.52 ÷ 244.60 = 7,707.
The file holds four errors. The next screens find them one at a time.
A split is not a crash
Tap the cliff in the line to look closer.
Holding Market 1, raw data or split-adjusted (simulated)
The account at each day's close. Orange: adjusted. Grey: as delivered. Dashed: the $10,000 start.
| Day | Close |
|---|
On day 1,501 the company made a 2-for-1 stock split: each share became two, each worth half: 100 shares at $425.31 became 200 at about $212.66, and nobody lost anything. The raw file keeps the old prices before the split, so a backtest sees a 50% fall. The fix, split adjustment, divides every earlier price by 2. Switch it on.
Adjusted, day 1,501 is an ordinary day: 211.54 ÷ (425.31 ÷ 2) − 1 = −0.52%. The first open becomes 244.60 ÷ 2 = 122.30, and holding turns $10,000 into 10,000 × 188.52 ÷ 122.30 = $15,415.
Dividends: the cash the prices forget
A cash dividend is money a company pays to whoever owns its shares. This one pays every 63 trading days (about every three months), 0.5% of the share price: 41 times in ten years. On the ex-dividend day, the first day a buyer no longer gets that payment, the price drops by about the dividend (that cash has left the company), on top of that day's ordinary market move, but the owner has the cash. The raw file holds prices only, so a backtest sees each drop and never the cash.
The fix is to count the total return: the price change plus the dividend. Credited means the cash buys more shares (it is reinvested). Switch dividends on.
Holding Market 1, split-adjusted, with and without dividends (simulated)
Coloured: the setting you picked. Grey: the other one. Dashed: the $10,000 start.
The first ex-dividend day was day 41. The close fell from $118.75 to $111.21, and the dividend was $0.59. Price only, the day moved (111.21 − 118.75) ÷ 118.75 = −6.35%; with the dividend, (111.21 − 118.75 + 0.59) ÷ 118.75 = −5.85%.
Many data sources credit dividends by adjusting prices: each price before an ex-dividend day is multiplied by (1 − the dividend ÷ the close before the ex-day), here 1 − 0.59 ÷ 118.75 = 0.995 for the first. Why this stands in for the cash: reinvesting the 0.59 buys about 0.5% more shares, and making every earlier price 0.5% lower lifts your return by about the same amount, with no shares to count. The 41 factors multiplied together come to 0.8151, so the first open becomes 122.30 × 0.8151 = $99.69, and holding ends at 10,000 × 188.52 ÷ 99.69 = $18,911.
"Bought at $99.69" is the first price rewritten in today's terms. You still paid $244.60 a share; only the ratio between prices matters.
A day that isn't there
A gap in data is a stretch with no prices (not the overnight gap between a close and the next open). SimMarket trades every weekday, with no holidays. Here are eleven rows around day 900, as the file has them, each with its close. Tap the first day after the hole.
Market 1's rows around day 900 (simulated)
Each tile is one row of the file: the weekday and its close.
Monday, day 901, is missing: the file jumps from Friday to Tuesday.
Most backtests read the file row by row (one row per day) without checking the calendar. So Tuesday's row shows the move since Friday's close, 144.56 ÷ 147.22 − 1 = −1.81%: two days of moves, read as one. A rule that counts rows ("hold 20 days") holds for 21 days when it spans the hole. Real markets do close on holidays, so check the exchange's calendar before calling a missing day an error.
A price that never was
A bad print is a price recorded wrong: a typing slip, a misplaced decimal point, a stray trade. The chart shows each day's move from one close to the next, with the split and dividends already fixed. Tap the move that looks wrong.
Market 1's daily moves, close to close, ten years (simulated)
One thin bar per day: up above the line, down below it. Orange band, once drawn: 20 times the usual size of a day's move (the middle value, or median, of the last 20 sizes), above and below, so it widens after wild days and narrows after calm ones; cut off at the chart's edge, ±25%.
Day 1,201's close reads $218.18 (prices here have the split and dividends already fixed, so they differ from the raw file). The true close was $181.82, between $181.01 the day before and $184.08 the day after. A price that jumps and comes straight back is the mark of a bad print.
The band compares each day's move with the usual size of the 20 moves before it (their median; a later screen shows how it is found), and flags any move more than 20 times that size. 20 clears every real day here: the biggest was 14.6 times its usual size. 2,585 days checked (the first 20 have nothing before them). It flagged two days: the bad print, and the day after, when the price comes back.
Which error costs more?
Now a rule that reacts to prices. The 10% exit rule: hold the shares, but when a day closes 10% or more below the day before, sell everything at the next open, and buy back at the open 20 rows later. On clean data no day falls 10%, so the rule just holds and ends at $18,911, like holding.
On the raw data, which error costs this rule most over ten years?
Fix them one at a time
The raw data again. Switch each fix on, in any order, and watch both results. Filled and Corrected take that day's true prices from a second source (here, SimMarket's clean data).
Two rules on Market 1, with the fixes you choose (simulated)
Coloured: the 10% exit rule. Grey, thicker, underneath: holding; where you see one line, the two are the same. Dashed: the $10,000 start.
Each error alone, with the other three fixed, against the exit rule's $18,911 on clean data:
| Error left in | Exit rule ends at | Cost |
|---|
The missing day cost nothing here because the exit rule never fired near it. The split costs most: a fake crash in the account, then a false exit. The missing dividends, 0.5% at a time, cost about twice as much as the bad print, 20% at once. The bad print's cost is also luck: its false exit sat out a rise. Holding never reacts to one day's move, so the bad print cost it nothing.
Adjusting prices for splits and dividends
Raw prices change units at a split and drop at each dividend. Adjusted prices rewrite the earlier prices so that every day's move is the real one. For a split, split adjustment divides every price before it by the split ratio. For a dividend, the day's total return counts the cash with the price change; the dividend factor F does the same by rewriting earlier prices: multiply every price before the ex-dividend day by F. The page's figures use the factor; with several dividends, multiply their factors together.
- P
- a price from before the split, as recorded
- s
- the split ratio: new shares per old share (a 4-for-1 split means s = 4; a reverse split, which merges shares, has s below 1)
- Padj
- that price in today's shares
- Pt, Pt−1
- day t's close and the previous day's close
- D
- the dividend per share, on its ex-dividend day (0 on other days)
- r
- the day's total return
- F
- the dividend factor, a little below 1
Split: day 1,500's close becomes 425.31 ÷ 2 = 212.655, so day 1,501 moved 211.54 ÷ 212.655 − 1 = −0.52%, not −50.26%.
Dividend: on day 41, r = (111.21 − 118.75 + 0.59) ÷ 118.75 = −5.85%, against −6.35% from prices alone. With the factor, F = 1 − 0.59 ÷ 118.75 = 0.995, so the earlier close becomes 118.75 × 0.995 = 118.16 and the day moved 111.21 ÷ 118.16 − 1 = −5.88%. The page's figures use this; the formula for r is a close approximation (−5.85% against −5.88%).
- the split ratios, dividend amounts and ex-dividend dates come from a reliable record;
- the dividend is cash (shares handed out instead need their own adjustment).
A rule reads price levels ("buy under $150"): adjusted prices before a split are prices nobody traded at. Or the record is wrong or missing (lesson D6, Cleaning and adjusting data).
Try it on new numbers. A share closes at $80.00. The next day is its ex-dividend day: it closes at $79.70 and pays a dividend of $0.60. What is that day's total return, in %?
Flagging a bad print: the usual range
A bad print is an outlier: a value far outside the usual range. To say what is usual, use the median: sort the numbers and take the middle one (with an even count, halfway between the middle two). One huge error drags an average a long way but barely moves a median, so the median is called robust. The check:
- rt
- day t's move from the close before, in %
- |rt|
- its size, with any minus sign dropped
- mt
- the usual size: the median of the sizes of the 20 moves before day t
- k
- how many times the usual size counts as suspicious (here 20, which clears the biggest real day, 14.6 times its usual size)
The 20 moves before it, in %: 0.25, −0.19, 0.68, −0.99, 0.49, −0.58, −0.35, 1.00, 0.85, 0.51, 0.40, 0.55, 0.09, −0.76, −0.18, −0.08, 0.25, 0.33, −0.03, 0.40. Sorted by size, the 10th and 11th are both 0.40, so m = 0.40% and the line is 20 × 0.40% = 8.0%. Day 1,201 moved +20.53%: flagged.
Next day, the window drops the first 0.25 and takes in the 20.53. The median only moves to (0.40 + 0.49) ÷ 2 = 0.445%, so the line is 8.9% and the −15.63% is flagged too. Their average would jump from 0.448% to 1.462%, putting the line at 29.24% and letting the −15.63% through.
- errors are rare, so most of the 20 days in each window are normal;
- an error is big next to a normal day's move.
A small error (2% off) hides among normal moves. A real crash is flagged too, so a flag is a question for a second data source, not a verdict (lesson A6, Fat tails). A missing day is no move at all: only the calendar finds it, by comparing the file's dates with the trading calendar (weekdays minus holidays).
Try it on new numbers, with a five-day window. The last five moves had sizes 0.3%, 0.5%, 0.4%, 0.9% and 0.6%. With k = 20, above what size is the next move flagged, in %?
What this means for your backtest
Four checks before you trust a result. Turn them over one at a time.
- Read returns from adjusted prices.
Split-adjusted, with dividends credited, unless the rule needs the price as traded.
- Question every big one-day move.
One that matches a split record is a split. One that comes straight back is probably a bad print.
- Check the dates against the calendar.
A missing day turns two days into one row and stretches every count of rows.
- Fix the data before you judge the rule.
Here the errors decided whether ten years of holding made $8,911 or lost $2,293.
A Friday that vanished
What should you do?
A 75% crash
What was the day's real move, in %?
Hint: first put the earlier close in today's shares (divide by the split ratio), then compare.
Three flags
What next?
Lesson done
Raw price data has traps. An unadjusted split looks like a crash, missing dividends make every holding return too low, a missing day turns two days into one row, and a bad print triggers trades on a price nobody traded. Which error matters most depends on the rule: here a quiet 0.5%, 41 times, cost the exit rule more than one close 20% too high, whose cost was partly luck.
Key ideas
- Raw data
- prices exactly as recorded, nothing corrected
- Stock split
- each share becomes s shares, each worth 1/s; nothing is lost
- Split adjustment
- divide every price before the split by the split ratio s
- Ex-dividend day
- the first day a buyer no longer gets the dividend; the price drops by about the dividend
- Total return
- the price change plus any dividend
- Gap (missing day)
- a stretch with no prices; the next row holds the moves of several days
- Bad print
- a price recorded wrong; it usually jumps and comes straight back
- Median, robust
- the middle value once sorted; robust because one huge error barely moves it
- Outlier
- a value far outside the usual range; a question for a second source
Formulas
Padj = P ÷ s · r = (Pt − Pt−1 + D) ÷ Pt−1 · F = 1 − D ÷ Pt−1 (multiply every earlier price by F) · flag if |rt| > k × median of the last 20 sizes