Course B · Backtesting Essentials

Glossary

5

5% line: a result a no-edge try reaches less than 5 times in 100; past it is called statistically significant. Fair for one test. B7 The best of many

A

Annualised Sharpe ratio: the Sharpe ratio per year: the daily one × √252, for independent days with the same average and volatility all year B12 The Sharpe ratio, and its error bar

B

Bad print: a price recorded wrong; it usually jumps and comes straight back B4 Data quality
Bar: one period's prices: open, high, low and close; complete only at its close B1 Inside a backtest engine
Benchmark: a simple alternative that a rule's result is measured against B13 Benchmarks
Bonferroni correction: to be fooled at most α of the time across the whole search (usually 0.05), raise the line so each of n tests clears it by luck only α ÷ n of the time B7 The best of many
Bootstrap: a run that draws the backtest's trades again with replacement; the end changes too B11 Monte Carlo

C

Confidence interval: the measured number ± z standard errors: a range worked out from one sample B5 Sampling error and confidence intervals
Confidence level: the share of samples whose interval catches the true value (95% is usual); a record of the method, not a chance for one interval B5 Sampling error and confidence intervals

D

Delisted: taken off the exchange: the stock stops trading B3 Survivorship bias

E

Edge: a real reason to win more often than a coin; with no edge (and wins and losses of equal size), a strategy wins half its trades on average B6 Hypothesis tests
Event-driven backtest: an engine that replays the data bar by bar; on bar t the rule reads bars 1 to t only B1 Inside a backtest engine
Ex-dividend day: the first day a buyer no longer gets the dividend; the price drops by about the dividend B4 Data quality
Excess: a rule's result minus holding's share, over the same days B15 A research workflow

F

False positive: a no-edge result that clears the line anyway B7 The best of many
Fan: the account lines of all the runs drawn together B11 Monte Carlo
Finding: a reported version that clears the line written in the plan B15 A research workflow
Forward test: a rule, unchanged, run on new days as they arrive B10 The held-out testalso in B16

G

Gap (missing day): a stretch with no prices; the next row holds the moves of several days B4 Data quality
Grid: the results of every version laid out by their settings B14 Robustness

H

Held-out data: the most recent stretch of data, set aside before any tuning or picking and left unseen until one chosen rule is run on it once B10 The held-out test
Held-out test: data kept aside while choosing, run once on the final version B7 The best of manyalso in B10
Holding's share: just holding's result times the share of days a rule is in the market: a yardstick that takes out a rising market's average rise B15 A research workflow
Hypothesis: a statement a test can count against: the rule, the data, the yardstick and the result that would make it wrong, written before any run B15 A research workflow
Hypothesis test: see what results the null hypothesis produces, then where yours falls B6 Hypothesis tests

I

Implementation shortfall: the difference between the price when you decide and the price you really get B1 Inside a backtest engine
In-sample: the stretch of past data used to tune a rule or to pick it from several (lesson B8's tuning data) B9 In-sample, out-of-sample and walk-forward
Independent trades: trades where one result doesn't change the next one's B11 Monte Carlo
Information set: everything known up to a moment; a rule may read only from it B2 Look-ahead bias

J

Just holding: buying the market at the start, never selling, paying for the one buy B13 Benchmarks

L

Lead: a rule's result minus just holding's result over the same days B9 In-sample, out-of-sample and walk-forwardalso in B13
Leakage test: scramble everything after each decision; if any decision changes, the rule read the future B2 Look-ahead bias
Line: the result a rule must clear to pass a test, fixed before the run B10 The held-out test
Look-ahead: a rule using anything from after bar t (lesson B2) B1 Inside a backtest engine
Look-ahead bias: a backtest that uses information which wasn't available at the moment of the decision B2 Look-ahead bias

M

Median: the middle of the sorted results: the 50th percentile B11 Monte Carlo
Median, robust: the middle value once sorted; robust because one huge error barely moves it B4 Data quality
Monte Carlo simulation: doing something random many times and counting what comes out; each time is one run B11 Monte Carlo
Multiple testing: trying many versions and keeping the best (also: data snooping). Each try is one more chance of a false positive. B7 The best of many

N

Neighbours: two versions one step apart in either setting or both, diagonals included: near copies that share most of their days B14 Robustness
New data: data that played no part in tuning a rule or in choosing it B8 Overfitting
Next-bar fill: the order fills at the next bar's open, the first price after the decision B1 Inside a backtest engine
Noise: the part of a past result that came from luck and has no reason to repeat B8 Overfitting
Null hypothesis: the dull explanation a test starts from: no edge, just a coin B6 Hypothesis tests

O

One-sided, two-sided: "does it beat a coin?" counts one side; "is it different from a coin?" counts both B6 Hypothesis tests
Order, fill: the instruction to trade, and the price it actually trades at B1 Inside a backtest engine
Out-of-sample: a later stretch of data that played no part in tuning a rule or in picking it (lesson B8's new data). Once its result is used to tune or to pick, it is in-sample B9 In-sample, out-of-sample and walk-forward
Outlier: a value far outside the usual range; a question for a second source B4 Data quality
Overfitting: tuning a rule until its settings fit the noise in the tuning data as well as anything real in it (also: curve fitting) B8 Overfitting
Overnight gap: the move from one close to the next open B1 Inside a backtest engine

P

p-value: the share of no-edge results at least as good as yours. Not the chance the strategy works. B6 Hypothesis tests
Peak: a lone high version with lower ones around it; it can be luck, and so can a plateau B14 Robustness
Percentage point: the gap between two percentages: +86.3% to +184.7% is 98.4 points B3 Survivorship bias
Percentage points: the plain difference of two percents B13 Benchmarks
Percentile: the p-th percentile of N sorted results is the one in place p × N ÷ 100 (rounded up), counted from the lowest; the rule holds for any sorted list B11 Monte Carlo
Plateau: a wide patch of similar results; neighbouring versions share most of their days, so a plateau can be luck too B14 Robustness
Plateau middle: the version with the highest neighbourhood average; still the best of many versions B14 Robustness
Point: one percentage point: 55% to 58% is 3 points B5 Sampling error and confidence intervals
Point-in-time universe: the list as it stood on each date, including stocks that later died B3 Survivorship bias

R

Raw data: prices exactly as recorded, nothing corrected B4 Data quality
Raw file, clean file: prices and volumes as recorded, and the same adjusted for events such as a split B17 Capstone: one idea, from question to verdict
Reshuffle: a backtest's own trades in a new random order; a fixed-amount account's end stays, its path and maximum drawdown change B11 Monte Carlo
Revised data: numbers corrected after they first came out; a backtest should use them as first published B2 Look-ahead bias
Risk-free rate: what money earns with next to no risk, such as interest on cash B12 The Sharpe ratio, and its error bar

S

Same-close fill: the order fills at the close the rule just read; optimistic B1 Inside a backtest engine
Sample-size calculation: working out how many days a test needs before an expected lead reaches the edge of its error bar B16 From backtest to forward test
Sampling error: the gap between a measured number and the true one, from which trades you happened to get B5 Sampling error and confidence intervals
Setting: anything in a trading rule that can be adjusted: a number of days, a level, an on/off switch B8 Overfitting
Sharpe ratio: the average return above the risk-free rate, divided by the standard deviation of the returns: return per unit of volatility B12 The Sharpe ratio, and its error bar
Shift by one bar: trade bar t on the signal from bar t − 1, so the fill comes after everything the rule read B2 Look-ahead bias
Shortcut: a step skipped or done after seeing results; in this lesson's simulation some added false findings only a little (no costs, the raw file) and some a lot (no untouched years, buying before the close was known) B17 Capstone: one idea, from question to verdict
Signal: the rule's decision to buy or sell, made at a bar's close B1 Inside a backtest engine
Smoothed returns: returns that each report an average over several days: the swings shrink, pushing the measured Sharpe ratio away from zero (up in 843 of 1,000 backtests in the lesson's simulation, down in 157) B12 The Sharpe ratio, and its error bar
Split adjustment: divide every price before the split by the split ratio s B4 Data quality
Standard error: how far luck typically moves a measured win rate: √(0.25 ÷ n) for a coin B6 Hypothesis tests
Standard error (SE): how far luck typically moves a measured number: its standard deviation across repeat backtests B5 Sampling error and confidence intervals
Standard error of a Sharpe ratio: how far luck typically moves a measured Sharpe ratio: about √((1 + S² ÷ 2) ÷ n) per period, for independent returns drawn from one bell curve and measured once B12 The Sharpe ratio, and its error bar
Statistically significant: a p-value under the significance level α, usually 5% (the 5% line). Fair for one test. B6 Hypothesis tests
Stock split: each share becomes s shares, each worth 1/s; nothing is lost B4 Data quality
Stop rule: a plan written before a forward test's first day: how long the test runs, and the result below which the rule is stopped B16 From backtest to forward test
Survivorship bias: the gap between a result on the survivors only and on everything that existed at the start B3 Survivorship bias

T

The four common leaks: using the same bar's close (the up-day rule); revised data; a longer bar that isn't finished; an indicator off by one row B2 Look-ahead bias
Total return: the price change plus any dividend B4 Data quality
Tuning: picking the settings that give a rule its best backtest on a chosen stretch of data, the tuning data B8 Overfitting
Tuning gap: a tuned rule's result on its tuning data minus its result, with the same settings, on new data of the same length B8 Overfitting

U

Universe: the list of stocks a test may choose from B3 Survivorship bias

V

Verdict: what a test measured, with its error bar, its years and its costs, and no more; "no finding" is not proof that nothing is there B17 Capstone: one idea, from question to verdict
Version: one full set of settings of a rule B8 Overfittingalso in B14
Version log: a record of every version run, kept or dropped; its count is the n in the chance of a lucky finding B15 A research workflow

W

Walk-forward: tuning a rule on a window of past data (the tuning window), testing it on the stretch right after (the test stretch), then moving both forward and repeating B9 In-sample, out-of-sample and walk-forward
Walk-forward record: a walk-forward's test stretches joined in order: out-of-sample as long as nothing was picked by looking at it B9 In-sample, out-of-sample and walk-forward

Z

z-test: a win rate's distance from 0.5 in standard errors, read off the bell B6 Hypothesis tests