Backtesting Essentials · Lesson 12
The Sharpe ratio, and its error bar
Return per unit of volatility puts two results on one scale. Measured on a short backtest, that number comes with a wide error bar.
Know the basics? Read the lesson notesSame return, different ride
A backtest runs a trading rule over past prices to see what it would have made. Here the rules are made up and a computer draws every day. Rules A and B each put $10,000 to work on every trading day of one year: 252 days. A day's return is what the day made, as a percent of that $10,000: +0.25% adds $25. Play the year.
Two rules, one year (simulated)
Colour: rule A. Dark: rule B. Dashed: the $10,000 start.
Both ended at $11,260: +12.6% for the year, or 12.6 ÷ 252 = +0.05% on the average day (they were made that way).
Volatility is the standard deviation of the returns: how far a typical day lands from the average day (lesson A8). A's daily volatility is 0.50% and B's 2.00%, four times as much. The return alone can't tell the two rules apart.
One scale for both
How much a rule makes depends on how much money it works with, which is a choice. With half of its $10,000 at work, B makes half as much every day: the year halves, and so does the volatility. Slide B's size down until its volatility matches A's.
Rule B at a smaller size (simulated)
Colour: rule A. Dark: rule B at the size you set. Grey: rule B at full size.
At a quarter of its size B swings as much as A, 0.50% a day, and made 3.15% where A made 12.6%.
Dividing the average return by the volatility says the same. A: 0.05 ÷ 0.50 = 0.10. B: 0.05 ÷ 2.00 = 0.025 at every size, because the size multiplies both alike. This is the Sharpe ratio, return per unit of volatility: one scale for rules of any size.
One thing comes off the return first: the risk-free rate, what money earns with next to no risk, such as interest on cash. This lesson sets it to 0: its returns are made up, so read each as a return above that rate already.
The Sharpe ratio
The Sharpe ratio is a rule's average return above the risk-free rate, divided by the standard deviation of its returns.
It is usually annualised: quoted per year. A year's return is the total of 252 days: 252 times the average day. If one day's return doesn't change the next day's, the standard deviation of that total is only √252 times one day's, because up days and down days partly cancel (a +1 and a −1 total 0: lesson A8). So the yearly ratio is 252 ÷ √252 = √252 ≈ 15.87 times the daily one.
For the precise
√252 is for daily returns. Other periods use the square root of their own count in a year: √12 for monthly returns.- μ
- "mu", the average return per period (a day, in this lesson)
- rf
- the risk-free rate over the same period (0 in this lesson)
- s
- the sample standard deviation of the returns, dividing by n − 1 (lesson A8): the volatility
- S
- the Sharpe ratio per period: a daily one from daily returns
- Syear
- the annualised Sharpe ratio, from daily returns that are independent, with the same average and volatility all year
Five days return +3%, −1%, +3%, −1% and +1%. Average: (3 − 1 + 3 − 1 + 1) ÷ 5 = 1%. Distances from it: +2, −2, +2, −2, 0. Their squares add up to 4 + 4 + 4 + 4 + 0 = 16. 16 ÷ (5 − 1) = 4, and √4 = 2%: the volatility. So S = (1 − 0) ÷ 2 = 0.5 a day.
- the risk-free rate covers the same period as the returns;
- for the yearly figure: the days are independent (one day's return doesn't change the next day's), with the same average and volatility all year.
One day's return depends on the day before (lesson C1, Time series basics): × √252 then gives the wrong yearly figure.
New numbers. A rule averaged +0.07% a day with a daily volatility of 1.2%, while cash earned 0.01% a day. What is its yearly Sharpe ratio?
One rule, five backtests
Now take a rule built so that its true Sharpe ratio, the one it would show over endless days, is known. Each day is drawn from a bell curve (most days near the average, few far out: lesson A5) with an average of +0.063% and a volatility of 1%, whatever the day before did. Its true daily Sharpe ratio is 0.063 ÷ 1 = 0.063, and its true yearly one 0.063 × 15.87 = 1.00.
A one-year backtest sees 252 such days and works out a Sharpe ratio from them. Run five.
One rule, one year at a time (simulated)
Colour: the latest backtest. Grey: the ones before it. Dashed: the $10,000 start.
Five backtests of one rule measured yearly Sharpe ratios of 2.1, 0.5, 0.4, 1.3 and 0.0. The rule never changed: its true Sharpe ratio was 1.00 every time.
The gap comes from which days a backtest happened to get: sampling error (lesson B5).
1,000 backtests of one rule
Guess first. In 1,000 one-year backtests of this rule, most of the measured Sharpe ratios will land within how far of the true 1.00?
1,000 one-year backtests of one rule (simulated)
Each column counts the backtests in a range 0.25 wide. Dashed: the true 1.00. Shaded: the band you picked.
You guessed within 0.1. That band held 74.
You guessed within 0.5. That band held 370.
You guessed within 2, the only band of the three that held most: 949.
The three bands held 74, 370 and 949 of the 1,000 backtests. The lowest measured −2.15 and the highest 4.48, and 168 measured a Sharpe ratio below zero for a rule whose true one is 1.00.
Their standard deviation is 1.03. The standard deviation of a measured number is its standard error (lesson B5): how far luck typically moves it.
Shorter and longer backtests
The same rule, in 1,000 backtests of 50 days and 1,000 of 2,000 days, about eight years. Guess first: of the 1,000 backtests of 50 days, how many measure a Sharpe ratio below zero? Then look at both lengths.
1,000 backtests of each length (simulated)
Dashed: the true 1.00.
You guessed about 30. It was 345.
You guessed about 150. It was 345.
You guessed about 350, the closest of the three. It was 345.
Over 50 days the standard error was 2.31, and 345 of the 1,000 backtests were below zero. Over 2,000 days it was 0.36, and 2 were below zero.
A longer backtest holds more days, so more of the luck cancels.
The standard error of a Sharpe ratio
You get one backtest, not 1,000. For independent returns drawn from one bell curve, the standard error of a measured Sharpe ratio follows from the Sharpe ratio and the number of returns. The error bar is lesson B5's 95% confidence interval: 1.96 standard errors either side, a range from a method that catches the true value about 95 times in 100.
- n
- the number of returns in the backtest (days, in this lesson)
- S
- the Sharpe ratio per period, not annualised: a yearly figure goes back to daily first, ÷ √252. The measured one stands in for the unknown true one
- SE
- the standard error of that per-period Sharpe ratio, for independent returns drawn from one bell curve and measured once
- SEyear
- the standard error of the annualised Sharpe ratio, from daily returns, under the same conditions
S = 0.063 a day and n = 252. 0.063² ÷ 2 = 0.002. (1 + 0.002) ÷ 252 = 0.003976, and √0.003976 = 0.0631. The yearly ratio is the daily one × 15.87, so its standard error is too: 0.0631 × 15.87 = 1.00, against the 1.03 you measured. With n = 50 the same sum gives 2.247 (measured: 2.31), and with n = 2,000 it gives 0.355 (measured: 0.36).
The S² ÷ 2 is there because the volatility is itself an estimate. For daily returns it is tiny, so SE is close to 1 ÷ √n: four times the days, half the bar.
- each return is independent of the ones before it;
- the returns are all drawn from one bell curve: the same average and volatility throughout;
- there are enough returns: the formula is approximate, and rougher for few;
- the backtest was run once, as planned, not picked as the best of several.
One day's return depends on the day before, as when each return is an average over several days (lesson C1, Time series basics). A rule has rare large losses (lesson A6, Fat tails). The backtest was picked from many, so it is not one measurement (lesson B7, The best of many). Two steps on, the lesson counts all three.
New numbers. A backtest of 400 days measured a daily Sharpe ratio of 0.08. What is the standard error of its yearly Sharpe ratio?
1.5 on 50 days, or 1.0 on 2,000?
Two rules, one backtest each. The first measured a yearly Sharpe ratio of 1.5 over 50 days, the second 1.0 over 2,000 days. Which is the better rule? Guess, then see their error bars.
Two measured Sharpe ratios with their 95% error bars
You guessed the first. These two numbers can't tell.
You guessed the second. These two numbers can't tell.
You guessed that these two numbers can't tell, and they can't.
The formula takes the daily Sharpe ratio, so the yearly 1.5 goes back to daily first: 1.5 ÷ 15.87 = 0.0945. Then 0.0945² ÷ 2 = 0.0045, (1 + 0.0045) ÷ 50 = 0.02009, √0.02009 = 0.1417, and 0.1417 × 15.87 = 2.249 a year. The error bar reaches 1.96 × 2.249 = 4.41 either side of 1.5: from −2.91 to 5.91.
If your own sum gives 3.27 for that standard error, you put the yearly 1.5 into the formula: divide by 15.87 first.
The second, the same way with n = 2,000: 0.355 a year, so its bar reaches 1.96 × 0.355 = 0.70: from 0.30 to 1.70.
The first bar holds zero, and the whole of the second bar. Now a count, for a rule with nothing in it: the same bell-curve days with an average of 0%, a true Sharpe ratio of 0.
A rule with nothing in it (simulated)
The rule with nothing in it measured 1.5 or more in 247 of its 1,000 backtests of 50 days, and 1.0 or more in 5 of its 1,000 backtests of 2,000 days.
So the two numbers do not show that the first rule is better: the 0.5 between them is far inside the first bar, and in this simulation a rule with nothing in it reached the first result far more often than the second. This is for independent, bell-curve days and one backtest of each rule.
For the precise
The second rule's sum in full: 1.0 ÷ 15.87 = 0.063 a day. 0.063² ÷ 2 = 0.002. 1.002 ÷ 2,000 = 0.000501, √0.000501 = 0.0224, and 0.0224 × 15.87 = 0.355. Standard errors that go into an error bar are carried to three decimals. Set side by side 1,000 times, a 50-day backtest of the rule with nothing in it measured a higher Sharpe ratio than a 2,000-day backtest of the rule with a true 1.00 in 323 pairs.Three ways to a high Sharpe ratio
A Sharpe ratio sees an average and a standard deviation, nothing else. In this simulation the rule with nothing in it, backtested once over one year, measured 2.0 or more in 28 of 1,000 backtests. Three things raise such a count. Look at all three.
1,000 one-year backtests of each (simulated)
Columns as before. Colour: measured 2.0 or more. Dashed: the true Sharpe ratio.
The best of 20. Backtest the rule with nothing in it 20 times and keep the best, as a search does (lesson B7). The kept backtests averaged 1.86, and 361 of the 1,000 measured 2.0 or more. Run over one more year, the kept ones averaged 0.03.
A rare large loss. On an ordinary day this rule makes +0.05% on average, with a volatility of 0.2%. On 1 day in 250, on average, it loses 12.45%, which makes its true average day 0%. In 338 of its 1,000 backtests the year held no such day, and those measured 3.98 on average. In all, 331 of the 1,000 measured 2.0 or more.
Smoothed returns. Take the rule with a true 1.00 and let each day report the average of its last five days' returns, for example when prices are updated late. The average day barely moves and the swings shrink: the smoothed backtests measured 2.21 on average, against 0.97 for the same days unsmoothed, and 517 of the 1,000 measured 2.0 or more, against 144 unsmoothed.
The error bar fails with them: where a 95% bar should hold the true value about 950 times in 1,000, it held the true 0 in 617 of the best-of-20 backtests and 668 of the rare-loss ones, and the rule's true 1.00 in 592 of the smoothed ones.
And a standard deviation counts days above and below the average alike, so the ratio cannot tell a large loss from a large gain of the same size.
For the precise
Around the plain one-year backtests of the rule with a true 1.00, the error bar held the true value in 946 of 1,000. The rare loss: 249 days in 250 are ordinary, a chance of 0.996, and 0.996 × 0.05 = 0.0498%. The loss takes 0.004 × 12.45 = 0.0498%, which leaves an average of 0%. All 331 rare-loss backtests that measured 2.0 or more were among the 338 with no large loss; the 662 with one or more averaged −0.34. A loss and a gain of the same size: five days of +2%, +2%, +2%, +2% and −3%, and five of 0%, 0%, 0%, 0% and +5%, both average 1% with a volatility of 2.24%, so both have a daily Sharpe ratio of 1 ÷ 2.24 = 0.45. The textbook names: a return that depends on the one before it is autocorrelation, and the smoothed days have it, because neighbours share four of their five days. More extreme days than a bell curve are fat tails, and extreme days that fall on one side are skew; the rare loss is both.What this means for your backtest
- A Sharpe ratio is a measured number.
Read it with the number of returns behind it and its error bar: about 2 either side for one year of independent, bell-curve days.
- Compare two Sharpe ratios by their bars.
If one bar holds the other number, the gap can be luck alone.
- Ask what the number can't see.
Versions tried, smoothed returns, a large loss that hasn't happened yet. For a picked backtest this lesson gives a count, not a correction: the bar held the true value in 617 of 1,000 best-of-20 backtests. The corrections for a short backtest and for versions tried, the Probabilistic and the Deflated Sharpe ratio (Bailey and López de Prado), are lesson C5's, Many tests at once.
"A Sharpe ratio above 3"
How far either side of the yearly Sharpe ratio does the 95% error bar reach?
Hint: SE ≈ √((1 + S² ÷ 2) ÷ n) with S = 0.2 and n = 63. Multiply by 15.87 for the yearly standard error, then by 1.96.
Your own two rules
What do the two numbers tell you?
Lesson done
The Sharpe ratio is a rule's average return above the risk-free rate, divided by the standard deviation of its returns. Measured on a backtest it is an estimate. For independent returns drawn from one bell curve and measured once, its standard error is about √((1 + S² ÷ 2) ÷ n) per period, so a short backtest gives a wide error bar: in this lesson's simulation a rule with a true Sharpe ratio of 1.00 measured from −2.15 to 4.48 over one year. Two Sharpe ratios can differ by luck alone.
Key ideas
- Sharpe ratio
- the average return above the risk-free rate, divided by the standard deviation of the returns: return per unit of volatility
- Risk-free rate
- what money earns with next to no risk, such as interest on cash
- Annualised Sharpe ratio
- the Sharpe ratio per year: the daily one × √252, for independent days with the same average and volatility all year
- Standard error of a Sharpe ratio
- how far luck typically moves a measured Sharpe ratio: about √((1 + S² ÷ 2) ÷ n) per period, for independent returns drawn from one bell curve and measured once
- Smoothed returns
- returns that each report an average over several days: the swings shrink, pushing the measured Sharpe ratio away from zero (up in 843 of 1,000 backtests in the lesson's simulation, down in 157)
Formulas
S = (μ − rf) ÷ s · Syear = S × √252
SE ≈ √((1 + S² ÷ 2) ÷ n) · SEyear = SE × √252
error bar = Syear ± 1.96 × SEyear
S in the standard error is per period. Every line needs independent returns with the same average and volatility throughout; the standard error also needs one bell curve and one backtest, not picked.