Course B · Backtesting Essentials

Formula sheet

B1 Inside a backtest engine

The backtest loop

Pfill = Ot+1
t
the bar on which the rule decides; its close is the last price the rule has read
Ot+1
the open of the next bar: the first price after the decision (Ot is bar t's open, Ct its close)
Pfill
the price the order fills at: a next-bar fill. A same-close fill would use Ct, the close the rule just read

B1 Inside a backtest engine

What a same-close fill assumes

rsame ≈ rnext + gbuy − gsell
rsame
the trade's result in %, with same-close fills
rnext
the same trade's result in %, with next-open fills
gbuy
the overnight gap after the buy decision, in %
gsell
the overnight gap after the sell decision, in %
≈
roughly equal: percentage moves multiply rather than add, but for moves of a few percent the difference is tiny

B2 Look-ahead bias

Use only what was known: shift by one bar

positiont = signalt−1
t
the bar you trade; the order fills at its open
signalt−1
the rule's answer (1 = buy, 0 = don't), worked out at the close of bar t − 1 from bars up to t − 1 only
positiont
whether you hold during bar t (1 = hold, 0 = don't)

B2 Look-ahead bias

Two tests for a leak

positiont(past, future) = positiont(past, future′)
positiont
the rule's decision for bar t, made at its open (1 = buy, 0 = don't): the decisions you scrambled
past
everything known at bar t's open
future, future′
what really happened after that moment, and a scrambled version of it

B3 Survivorship bias

Survivorship bias, measured

bias = R̄S − R̄all = d × (R̄S − R̄D)
R̄S
the average return of the survivors (the stocks still trading at the end)
R̄all
the average return of everything that existed at the start
R̄D
the average return of the dead ones (delisted), up to their last price
d
the share that died: the number delisted ÷ the number at the start

B3 Survivorship bias

The fix: the list as it stood

R̄all = (NS × R̄S + ND × R̄D) ÷ (NS + ND)
NS, ND
how many survived, how many died
R̄S, R̄D
their average returns, the dead ones' up to their last price
R̄all
the honest average: what the plan made on the list as it stood at the start

B4 Data quality

Adjusting prices for splits and dividends

Padj = P ÷ s
r = (Pt − Pt−1 + D) ÷ Pt−1
F = 1 − D ÷ Pt−1
P
a price from before the split, as recorded
s
the split ratio: new shares per old share (a 4-for-1 split means s = 4; a reverse split, which merges shares, has s below 1)
Padj
that price in today's shares
Pt, Pt−1
day t's close and the previous day's close
D
the dividend per share, on its ex-dividend day (0 on other days)
r
the day's total return
F
the dividend factor, a little below 1

B4 Data quality

Flagging a bad print: the usual range

flag day t if |rt| > k × mt
rt
day t's move from the close before, in %
|rt|
its size, with any minus sign dropped
mt
the usual size: the median of the sizes of the 20 moves before day t
k
how many times the usual size counts as suspicious (the lesson uses 20)

B5 Sampling error and confidence intervals

How wide the bar is: the standard error

SE of an average = σ ÷ √n
SE of a win rate = √(p(1 − p) ÷ n)
SE
the standard error: the standard deviation of the measured number across repeat backtests
σ
the standard deviation of one trade's result (σ is the Greek letter sigma)
n
the number of trades
p
the win rate as a fraction (55% = 0.55). The true one is unknown, so in practice the measured one stands in for it

B5 Sampling error and confidence intervals

The confidence interval

estimate ± z × SE
estimate
the number you measured: a win rate, an average trade
z
standard errors either side, set by the confidence level: 1.28 for 80%, 1.64 for 90%, 1.96 for 95%, 2.58 for 99%
SE
its standard error, from the last card

B6 Hypothesis tests

The null hypothesis and the p-value

p-value = (no-edge runs at least as good as yours) ÷ (all no-edge runs)
significant when p-value < α
no-edge runs
results the null hypothesis produces: here, a coin with the same number of trades
α
the significance level, chosen before the test: usually 0.05, the 5% line

B6 Hypothesis tests

A shortcut: the z-test for a win rate

z = (p̂ − 0.5) ÷ √(0.25 ÷ n)
p̂
"p-hat": your win rate as a decimal (54.4% is 0.544)
0.5
a coin's average win rate
0.25
0.5 × 0.5: a coin's chance to win times its chance to lose (the p × (1 − p) in the previous lesson's standard error, with p = 0.5)
n
the number of trades
z
how many standard errors the win rate sits above 0.5

B7 The best of many

The chance that at least one no-edge try passes

at least one clears: 1 − (1 − p)n
raised line, for each test: peach ≈ α ÷ n
p
the chance one no-edge try clears the line (0.0438 for 139 wins of 250)
n
the number of tries: every version you tested, kept or dropped
(1 − p)n
the chance all n miss
α
the most you accept being fooled across the whole search (usually 0.05)
peach
the chance a no-edge try clears the raised line

B8 Overfitting

The tuning gap

tuning gap = tuned − new
tuned
a rule's result on the data its settings were tuned on
new
the same rule's result, with the same settings, on new data of the same length from the same market
tuning gap
the part of a tuned result that was not there on new data. A large tuning gap is a sign of an overfit rule

B8 Overfitting

Every setting multiplies the versions

versions = c1 × c2 × … × ck
k
the number of settings a rule has
c1, c2, …, ck
how many values the first setting can take, the second, and so on to the last (2 for an on/off setting)

B9 In-sample, out-of-sample and walk-forward

In-sample, out-of-sample and the lead a year

lead a year = (rule − hold) ÷ years
rule
a rule's result on a stretch of days
hold
just holding's result on the same days
years
the length of the stretch, in years
lead a year
the rule's lead over just holding for each year of the stretch (in the lesson: dollars on $100 held a day, before costs); below zero means behind

B9 In-sample, out-of-sample and walk-forward

Counting a walk-forward

test stretches = (N − T) ÷ S, rounded down
N
the length of all the data
T
the length of the tuning window, which stays the same as the window moves
S
the length of one test stretch, which is also how far the window moves at each step. N, T and S are in the same unit, such as years
rounded down
a last stretch shorter than S is not used, so the record is test stretches × S long

B10 The held-out test

The error bar on a held-out lead

error bar = lead ± 1.96 × σ × √n
lead
a rule's result minus just holding's over all the held-out days: the total, not the lead a year
σ
the standard deviation of one day's lead, measured in the held-out run itself: how far a day's lead typically sits from the average day's (σ is the Greek letter sigma)
n
the number of held-out days, so σ × √n is the standard error of the lead
1.96
standard errors either side for a 95% interval: 95 in 100 such intervals catch the true value, which is the rule's real long-run lead

B11 Monte Carlo

The percentile of a Monte Carlo

k = p × N ÷ 100
N
the number of runs
p
the percentile wanted: 50 for the median, 95 for the 95th
k
the place of that percentile among the N results sorted from lowest to highest (round up if not whole). At most N − k results are higher

B12 The Sharpe ratio, and its error bar

The Sharpe ratio

S = (μ − rf) ÷ s
Syear = S × √252
μ
"mu", the average return per period (a day, in this lesson)
rf
the risk-free rate over the same period (0 in this lesson)
s
the sample standard deviation of the returns, dividing by n − 1 (lesson A8): the volatility
S
the Sharpe ratio per period: a daily one from daily returns
Syear
the annualised Sharpe ratio, from daily returns that are independent, with the same average and volatility all year

B12 The Sharpe ratio, and its error bar

The standard error of a Sharpe ratio

SE ≈ √((1 + S² ÷ 2) ÷ n)
SEyear = SE × √252
error bar = Syear ± 1.96 × SEyear
n
the number of returns in the backtest (days, in this lesson)
S
the Sharpe ratio per period, not annualised: a yearly figure goes back to daily first, ÷ √252. The measured one stands in for the unknown true one
SE
the standard error of that per-period Sharpe ratio, for independent returns drawn from one bell curve and measured once
SEyear
the standard error of the annualised Sharpe ratio, from daily returns, under the same conditions

B13 Benchmarks

The lead over holding

lead = net − hold
net
the rule's result after costs over the test days, in % = (end value ÷ start value − 1) × 100
hold
just holding's result on the same market over the same days, after its one buy, in % = (end value ÷ start value − 1) × 100
lead
percentage points ahead of holding; a minus sign means behind

B14 Robustness

The neighbourhood average

N = (R1 + R2 + … + Rm) ÷ m
R1 … Rm
the results of a version and of every version next to it in the grid (leads a year, in this lesson)
m
how many of those are in the grid: 9 for a version inside, 6 on an edge, 4 in a corner
N
the version's neighbourhood average, for versions all run on the same days

B15 A research workflow

The version log and the chance of a lucky finding

chance at least one clears = 1 − (1 − p)n
p
the chance one version with nothing in it clears the line: 0.025 for z above 1.96, when its excess averages $0
n
the number of versions in the log, every one run, kept or dropped
(1 − p)n
the chance all n miss: 1 − p multiplied by itself n times

B16 From backtest to forward test

How many days a forward test needs

n = (1.96 × σ ÷ d)²
n
the number of forward days at which a lead equal to its average, d × n, reaches the edge of its 95% error bar
d
the rule's true average lead a day. It is unknown: a picked backtest's figure is flattered, so n from it comes out too small
σ
the standard deviation of one day's lead
1.96
the multiplier for a 95% error bar (lessons B5 and B10)

B17 Capstone: one idea, from question to verdict

The arithmetic of a verdict

excess = R − f × H
error bar = excess ± 1.96 × σ × √n
z = excess ÷ (σ × √n); a finding when z > 1.96
R
the rule's result over the test days, after costs
H
just holding's result over the same days
f
the share of those days the rule is in the market
σ
the standard deviation of one day's excess (its typical distance from its average)
n
the number of test days; σ × √n is the standard error
1.96
the multiplier for a 95% interval