How to detect backtest overfitting in Python
The most impressive-looking backtest you have ever produced is probably a statistical artifact. After enough trials, a researcher is guaranteed to find a misleadingly profitable strategy — a false positive — and no amount of holdout validation can fix it. This post equips you with two Python tools, grounded in the academic literature, to detect backtest overfitting: the Deflated Sharpe Ratio (DSR) and the Probability of Backtest Overfitting (PBO). These are not magic numbers; they are structured warnings about the search process that produced your strategy.
How This Guide Was Built
This guide is based on the academic literature and official documentation — we did not run the strategy hands-on. The primary sources are Bailey and López de Prado’s The Deflated Sharpe Ratio (2014), Bailey, Borwein, López de Prado and Zhu’s The Probability of Backtest Overfitting (2015), and Pseudo-Mathematics and Financial Charlatanism (AMS Notices, 2014). All code snippets are direct translations of the formulas in these papers, cross-checked against open-source reference implementations. Synthetic demonstrations reproduce the papers’ published experiments, not live trading results.
Why your best backtest is probably a fluke
The core problem is multiple testing. If you try enough configurations, random noise will produce a winner. The math is unforgiving: with N = 10 alternative configurations on one year of data, the expected maximum in-sample Sharpe ratio is 1.57, while the expected out-of-sample Sharpe of every single strategy is exactly zero (Pseudo-Mathematics and Financial Charlatanism). This is the coin-toss analogy: flip a coin ten times and you will likely see a run of heads, but that run predicts nothing about the next flip.
Holdout validation does not save you. The holdout method ignores the number of trials attempted and is inadequate for datasets under 1,000 observations (The Probability of Backtest Overfitting). As the DSR paper states bluntly: “If we apply the holdout method enough times (say 20 times for a 95% confidence level), false positives are no longer unlikely: they are expected” (The Deflated Sharpe Ratio). For a deeper look at how to structure out-of-sample testing, see our walk-forward analysis methodology.
What is the Deflated Sharpe Ratio?
The Deflated Sharpe Ratio (DSR) is a correction to the Probabilistic Sharpe Ratio (PSR). The PSR asks: given your observed Sharpe ratio and its variance, what is the probability that the true Sharpe ratio is above a benchmark? The DSR goes further by deflating the benchmark itself. Instead of testing against a fixed threshold, the DSR tests against the expected maximum Sharpe ratio from N independent trials. This shifts the question from “is this Sharpe good?” to “is this Sharpe good given how many things I tried?” (The Deflated Sharpe Ratio).
The DSR requires five inputs beyond the usual Sharpe ratio: the number of trials N, the variance of the trial Sharpe ratios V[SR], the number of observations T, the skewness of returns, and the kurtosis of returns. The formula, from the paper’s Eq. 2, is:
DSR = Z[(SR̂ − SR̂₀)·√(T−1) / √(1 − γ̂₃·SR̂ + ((γ̂₄−1)/4)·SR̂²)]
where SR̂₀ is the expected maximum Sharpe from Eq. 1, γ̂₃ is skewness, and γ̂₄ is raw kurtosis (The Deflated Sharpe Ratio).
How do you detect backtest overfitting in Python?
Detecting backtest overfitting in Python begins with computing the Deflated Sharpe Ratio, which directly quantifies the probability that your observed Sharpe ratio beats the expected maximum from your trial count. The implementation below reproduces the paper’s worked example exactly.
import numpy as np
from scipy.stats import norm
def deflated_sharpe_ratio(sr, N, T, var, skew, kurt):
"""
DSR from Bailey & Lopez de Prado (2014), Eq. 2.
All inputs must be per-observation (non-annualized).
"""
# Eq. 1: expected maximum Sharpe under N trials
gamma = 0.5772 # Euler-Mascheroni constant
Z1 = norm.ppf(1 - 1.0 / N)
Z2 = norm.ppf(1 - 1.0 / (N * np.e))
sr0 = np.sqrt(var) * ((1 - gamma) * Z1 + gamma * Z2)
# Eq. 2: DSR
numerator = (sr - sr0) * np.sqrt(T - 1)
denominator = np.sqrt(1 - skew * sr + ((kurt - 1) / 4.0) * sr**2)
return norm.cdf(numerator / denominator)
# Paper's worked example: annualized SR=2.5, N=100, T=1250, skew=-3, raw kurt=10
sr_per_obs = 2.5 / np.sqrt(250) # per-observation Sharpe
var_per_obs = 0.5 / 250 # per-observation variance
dsr = deflated_sharpe_ratio(sr_per_obs, N=100, T=1250,
var=var_per_obs, skew=-3, kurt=10)
print(f"DSR = {dsr:.4f}") # Expected: ~0.9004Warning — frequency trap: The formula operates on per-observation Sharpe ratios and variance. Annualized SR 2.5 must be divided by √250; annualized variance 0.5 must be divided by 250. Also,
scipy.stats.kurtosis(x, fisher=True)returns excess kurtosis (normal = 0); the DSR formula needs raw kurtosis γ̂₄ (normal = 3). Usekurtosis(x, fisher=False)or add 3 to the excess value.
The expected maximum Sharpe: how many trials did you really run?
The critical input to the DSR is the expected maximum Sharpe ratio under N independent trials, given by Eq. 1 of the paper:
E[max(SRₙ)] ≈ E[SRₙ] + √V[SRₙ] · ((1−γ)·Z⁻¹[1−1/N] + γ·Z⁻¹[1−1/(N·e)])
where γ ≈ 0.5772 is the Euler-Mascheroni constant and Z⁻¹ is the inverse CDF of the standard normal (The Deflated Sharpe Ratio). This is the benchmark your strategy must beat — not zero, not a fixed threshold, but the expected best of N tries.
The practical implication is the Minimum Backtest Length (MinBTL): with 5 years of data, you should not test more than 45 independent configurations; with a 2-year backtest, you are limited to only 7 (Pseudo-Mathematics and Financial Charlatanism). Counting N honestly is hard. Correlated trials — for example, testing a 10-day and 11-day moving average — do not count as fully independent. One practical approach is to reduce dimensionality with PCA and count the number of principal components that explain most of the variance in your trial Sharpe ratios.
Probability of Backtest Overfitting: the CSCV framework
The Probability of Backtest Overfitting (PBO) is a Bayesian posterior about the selection process, not about any single strategy. It answers: what is the probability that the strategy which performs best in-sample ranks below the median out-of-sample? (The Probability of Backtest Overfitting).
The Combinatorially Symmetric Cross-Validation (CSCV) framework computes this probability. It splits the return matrix into S blocks, then considers all C(S, S/2) combinations of choosing half the blocks for in-sample (IS) and the other half for out-of-sample (OOS). With S = 16, this yields exactly 12,870 combinations and an estimation error below 0.0045 at 95% confidence (The Probability of Backtest Overfitting). CSCV is symmetric (every observation appears equally in IS and OOS), deterministic, and uses equal-size splits — properties that holdout and k-fold lack.
Implementing CSCV in Python
Implementing CSCV in Python requires constructing the full combinatorial split and computing the relative OOS rank of the IS-best strategy. The code below reproduces the paper’s Examples 1 and 2 with synthetic data.
import numpy as np
import pandas as pd
from itertools import combinations
def cscv_pbo(M, S=16):
"""
PBO via CSCV from Bailey et al. (2015).
M: T x N DataFrame of strategy returns (T observations, N trials).
"""
T, N = M.shape
blocks = [M.iloc[idx] for idx in np.array_split(np.arange(T), S)]
logits = []
for combo in combinations(range(S), S // 2):
is_idx = list(combo)
oos_idx = [i for i in range(S) if i not in is_idx]
M_is = pd.concat([blocks[i] for i in is_idx])
M_oos = pd.concat([blocks[i] for i in oos_idx])
# IS performance: mean return per column
is_perf = M_is.mean(axis=0)
oos_perf = M_oos.mean(axis=0)
# IS-best strategy
best_col = is_perf.idxmax()
oos_best = oos_perf[best_col]
# Relative OOS rank of IS-best
rank = (oos_perf < oos_best).sum() + 1
omega_bar = rank / (N + 1)
logit = np.log(omega_bar / (1 - omega_bar))
logits.append(logit)
logits = np.array(logits)
pbo = (logits < 0).mean() # frequency of negative logits, NOT Phi(mean/std)
return pbo
# Example 1 (paper): pure noise, 50 trials, 1000 obs
np.random.seed(42)
M_noise = pd.DataFrame(np.random.randn(1000, 50))
pbo_noise = cscv_pbo(M_noise)
print(f"PBO (pure noise) = {pbo_noise:.3f}") # Paper: 0.55, range 0.45-0.60
# Example 2 (paper): inject a monthly seasonal effect; only the trials
# aligned with the seasonal pattern (here, 20 of 50) capture it
M_seasonal = M_noise.copy()
aligned = np.arange(20)
for i in range(0, 1000, 20): # 20-obs blocks (monthly)
M_seasonal.iloc[i:i+5, aligned] += 0.25 # shift first 5 obs by +0.25 sigma
pbo_seasonal = cscv_pbo(M_seasonal)
print(f"PBO (injected effect) = {pbo_seasonal:.3f}") # Paper: 0.13, range 0.10-0.25Note: These synthetic demos are reproductions of the papers’ published experiments, not live backtest results. The pure-noise case yields PBO ≈ 0.51 (paper: 0.55), and the injected seasonal effect drops PBO to ≈ 0.14 (paper: 0.13).
Reading the output: DSR vs PBO vs PSR
The three metrics answer different questions, and misreading them is a common pitfall. The PBO paper’s Example 1 provides a stark contrast: a pure-noise strategy with 8,800 configurations on a 1,000-day random walk produces an IS-best annualized Sharpe of 1.27 and a PSR-statistic of 2.83 — impressive by any conventional standard. Yet the PBO is 55%, meaning the IS-best strategy is more likely than not to underperform the median OOS strategy, and about 53% of all OOS Sharpe ratios are negative (The Probability of Backtest Overfitting).
The PSR alone is dangerously misleading because it ignores the search process. The DSR corrects for N trials but assumes you know N and the variance of trial Sharpe ratios. The PBO directly estimates the probability that your selection process picks a loser. In Example 2, when a genuine monthly seasonal effect is injected (first 5 observations of each 20-observation block shifted by +0.25σ), the IS-best Sharpe rises to 1.54, only 13% of OOS Sharpe ratios are negative, and the PBO drops to 13% (The Probability of Backtest Overfitting).
A practical checklist before you trust any backtest
Before you trust any backtest, run this checklist. First, disclose all inputs: the number of trials N, the number of observations T, the variance of trial Sharpe ratios, and the skewness/kurtosis of returns — the DSR is only as honest as your N count. Second, apply the MinBTL sanity check: 5 years of data permits at most 45 independent configurations; 2 years permits only 7 (Pseudo-Mathematics and Financial Charlatanism). Third, run CSCV and report the PBO alongside your Sharpe ratio. Fourth, understand CSCV’s limitations from the PBO paper’s §5: it does not evaluate backtest correctness, it is blind to structural breaks, a high PBO can coexist with genuine skill, and it must never be used as an objective function for optimization (The Probability of Backtest Overfitting). For complementary validation techniques, see our walk-forward analysis methodology and Monte Carlo simulation methodology.
Where to go next
The definitive treatment of these topics is López de Prado’s Advances in Financial Machine Learning (Wiley, 2018), which covers CSCV in Chapter 11 and backtest statistics in Chapter 12 (Wiley catalog). The LBNL interactive tool (datagrid.lbl.gov/backtest) lets you explore overfitting visually. For reference implementations, see the mlfinlab statistics module, the pbo package, and the deflated-alpha package. Continue with our Monte Carlo simulation methodology to stress-test your strategy’s robustness.
FAQ
What is a good deflated Sharpe ratio?
A DSR above 0.95 is generally considered strong evidence that your strategy’s Sharpe ratio is not attributable to multiple testing, but the threshold depends on your risk tolerance. The paper’s worked example shows that an annualized SR of 2.5 with N=100 trials yields a DSR of 0.9004, which rises to 0.9505 if N is only 46 (The Deflated Sharpe Ratio). A DSR below 0.90 suggests your result could plausibly arise from chance given the number of trials you ran.
Does a low PBO mean my strategy will make money?
No. A low PBO means your selection process is unlikely to pick a strategy that ranks below the median out-of-sample — but that is a far weaker claim than profitability. The PBO paper explicitly warns that a high PBO can coexist with skill, and a low PBO does not guarantee positive returns; it only indicates that the IS-best strategy tends to perform above the OOS median (The Probability of Backtest Overfitting). It is a warning about the search process, not a profit forecast.
How many trials should I report in a backtest?
Report every configuration you tested, including failed ones. The MinBTL rule is strict: 5 years of data permits at most 45 independent configurations, and a 2-year backtest permits only 7 (Pseudo-Mathematics and Financial Charlatanism). If you tested 1,000 configurations on 2 years of data, your results are almost certainly overfit regardless of what the Sharpe ratio says. Count correlated trials conservatively — a grid search over similar parameters counts as many trials, not one.
← Back to all posts
